Artificial intelligence-based radar and vision fusion intelligent monitoring and early warning method and device for debris flow

Abstract

The application discloses a method and equipment for intelligent monitoring and early warning of debris flow based on artificial intelligence, which comprises the following steps: collecting multi-modal monitoring data in a debris flow monitoring area; performing modal accessibility detection on the multi-modal monitoring data, generating an accessibility label graph, and labeling the inaccessible area of the main modal data; constructing a modal mapping atlas to determine a set of alternative modal paths; inputting into an improved DANN model to output the reconstruction representation of the main modal features and the corresponding alternative confidence score; performing a jump modal reconstruction operation to generate a main modal feature mask graph; using a spatial weight adjustment mechanism to fuse and process the original main modal features and the completed features to generate a disaster state vector; constructing a risk fluctuation trend curve to generate corresponding debris flow early warning instructions. The application improves the disaster recognition robustness and response efficiency in complex terrain and multi-obstacle interference environment, and has a wide application prospect.

Description

Technical Field

[0001] This invention relates to the field of geological disaster monitoring technology, and in particular to an intelligent monitoring and early warning method and equipment for debris flows based on artificial intelligence and fusion of radar and vision. Background Technology

[0002] With the increasing demand for geological disaster monitoring and early warning, especially in mountainous, canyon, and frequently rainy areas, intelligent sensing and real-time early warning technologies for sudden disasters such as debris flows have received widespread attention. Existing debris flow monitoring systems typically rely on single-source data channels, such as video image monitoring, radar ranging systems, or hydrological sensor networks, to determine the disaster risk status by identifying characteristic parameters such as slope movement, changes in surface elevation, or fluctuations in mud and water flow velocity. However, in complex field environments, these signal channels are highly susceptible to weather changes, terrain obstruction, or sensor malfunctions, leading to missing key data or reduced accuracy.

[0003] Video images often fail to capture effective footage in low-visibility environments such as heavy rainfall, nighttime, or dense fog, leading to severe degradation or even complete loss of image features. While radar signals possess penetrating properties, their sensitivity to dramatic changes in surface structure is insufficient, making it difficult to accurately detect localized mudslide accumulation and flow trends. Although hydrological parameters can reflect the overall trend of mud and water movement, their response is typically delayed, making it difficult to capture the initial state of a disaster in a timely manner. Existing methods often fail to quickly switch to effective alternatives when encountering main data channel anomalies or a decrease in signal-to-noise ratio, resulting in insufficient overall system robustness, delayed early warning command output, and severely restricting disaster response efficiency and early warning reliability.

[0004] Therefore, how to provide an AI-based intelligent monitoring and early warning method and equipment for debris flow that integrates radar and vision is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose an intelligent monitoring and early warning method and device for debris flows based on artificial intelligence and integrating radar ranging, video images, hydrological observation, and infrared thermal imaging monitoring data. This invention fully integrates radar ranging, video images, hydrological observation, and infrared thermal imaging monitoring data, constructs a modal reachability detection and alternative feature reconstruction mechanism, uses an improved DANN model for main modal feature completion, and generates debris flow early warning commands by combining spatial fusion strategies and risk fluctuation analysis. It has the advantages of strong perception robustness, high adaptability, and excellent early warning accuracy.

[0006] The intelligent monitoring and early warning method for debris flow based on radar-visual fusion according to an embodiment of the present invention includes the following steps:

[0007] Step 1: Collect multimodal monitoring data within the debris flow monitoring area;

[0008] Step 2: Based on the established signal quality assessment criteria, perform modal reachability detection on the multimodal monitoring data, generate an reachability label map, and mark the inaccessible areas of the main modality data in the reachability label map;

[0009] Step 3: Construct a modality mapping map, determine the set of alternative modal paths based on the inaccessible regions, and extract the corresponding candidate alternative features;

[0010] Step 4: Input the candidate alternative features into the improved DANN model, and output the reconstructed representation of the main modality features and the alternative confidence score;

[0011] Step 5: Based on the alternative confidence score and the preset confidence threshold, perform a skip modality reconstruction operation to fill in the missing main modality features in the inaccessible region and generate a main modality feature mask map;

[0012] Step 6: Combining the aforementioned master modality feature mask, a spatial weight adjustment mechanism is used to fuse the original master modality features and the completed features to generate a disaster state vector;

[0013] Step 7: Construct a risk fluctuation trend curve based on the disaster state vector, identify the time period when the risk fluctuation level is greater than the preset fluctuation threshold, and generate the corresponding debris flow early warning instruction.

[0014] Optionally, step one specifically includes:

[0015] A radar ranging sensor module is deployed within the target area to collect data on changes in ground elevation and distances to obstacles, generating a radar ranging data sequence; a video image acquisition module is deployed to collect a continuous sequence of image frames, generating a video image data sequence; a hydrological information acquisition module is deployed to collect and calculate data on debris flow velocity, flow rate, and sediment content, generating a hydrological observation data sequence; and an infrared thermal imaging acquisition module is deployed to use a long-wave uncooled thermal imaging device in an infrared thermal imager to collect a sequence of surface temperature distribution images, identify the temperature difference distribution between water and solid particles in the debris flow, and generate an infrared image data sequence.

[0016] Time synchronization and spatial alignment processing are performed on radar ranging data sequences, video image data sequences, hydrological observation data sequences, and infrared image data sequences to unify the timestamps and spatial indexes of each modality of data and generate standardized multimodal monitoring data.

[0017] Optionally, step two specifically includes:

[0018] A modal data quality scoring function is constructed to calculate the signal-to-noise ratio, data missing rate, and modal continuity score for radar ranging data sequences, video image data sequences, hydrological observation data sequences, and infrared image data sequences, respectively.

[0019] Based on the established signal quality assessment criteria and the output of the modal data quality scoring function, the reachability status of each mode at each time step is marked.

[0020] Arrange the reachability states by time steps to generate an reachability label map, and assign a corresponding label channel to each mode;

[0021] The video image data sequence is set as the main modality data. According to the reachability label map, regions with a signal-to-noise ratio less than a preset threshold or missing data within a continuous time step in the video image channel are marked as main modality missing regions, and a main modality data unreachable region label map is generated.

[0022] Optionally, step three specifically includes:

[0023] A modal mapping map is constructed based on historical multimodal monitoring data. The modal mapping map includes the mapping relationship structure and joint distribution similarity index between radar ranging channels, video image channels, hydrological observation channels and infrared image channels. The mapping relationship structure establishes a set of mapping paths between modes based on space-time indexing rules. The joint distribution similarity index forms a similarity scoring matrix by calculating the overlap of feature distributions of each modal channel at the same spatial location and time step.

[0024] Using the unreachable region label map of the main modality data as input, the set of mapping paths whose spatial locations coincide with the missing regions of the main modality are retrieved in the modality mapping map;

[0025] Alternative modal paths with confidence scores greater than a preset confidence threshold are selected from the mapping path set to form an alternative modal path candidate set;

[0026] Based on the alternative modal path candidate set, the corresponding reachable modal data index is determined, and feature vectors that satisfy the index constraints are extracted from the radar ranging data sequence, hydrological observation data sequence and infrared image data sequence to form candidate alternative features.

[0027] Optionally, step four specifically includes:

[0028] An improved DANN model is constructed; the improved DANN model includes a multi-channel modal encoder, a modal confidence estimator, an adaptive adversarial module, and a skip modal reconstruction module;

[0029] The candidate substitution features are input into the multi-modal encoder, and low-dimensional feature representations are extracted from different modal channels respectively;

[0030] The low-dimensional feature representation is input into the modality confidence estimator to generate confidence scores for each alternative modality;

[0031] The low-dimensional feature representation and the confidence score are input into the adaptive adversarial module to perform feature domain alignment processing and generate aligned modality joint features.

[0032] The modal joint features are input into the skip modality reconstruction module, which outputs the reconstructed representation of the main modality features and generates an alternative confidence score corresponding to the reconstructed representation.

[0033] Optionally, the improved DANN model is specifically:

[0034] A multi-modal encoder is constructed to receive candidate substitution features from three alternative modal channels, and low-dimensional modal embedding representations of the three modalities are extracted through three independent convolutional coding networks respectively.

[0035] A modal confidence estimator is constructed by performing linear mapping and normalization on the low-dimensional modal embedding representations of the three modalities respectively to generate three confidence vectors. Based on each confidence vector and a preset confidence threshold, a modal dynamic weighting factor is constructed. The modal dynamic weighting factor includes a modal significance component and a modal stability component.

[0036] The modal saliency component is constructed based on the feature mean and variance in the modal embedding representation, and the saliency of each modality is jointly characterized by the feature response amplitude and the feature space distribution density. The corresponding modal saliency component is generated by calculating the average activation value of each channel in the embedding space and the variance in the embedding dimension, and combining it with the saliency scoring function. The modal stability component is constructed based on the confidence variation range and volatility of the modality within a historical time window, and the modal stability is quantified by the maximum confidence difference and standard deviation within the sliding window. The corresponding modal stability component is generated by calculating the difference between the maximum and minimum values ​​and the standard deviation of the three confidence vectors within the time window, and inputting them into the stability scoring function.

[0037] An adaptive adversarial module is constructed, which receives three modality embedding representations and corresponding modality dynamic weighting factors, performs cross-modality feature distribution alignment operation, optimizes the distribution consistency between the source modality and the target modality through a gradient inversion structure and a modality domain discriminant network, and generates joint modality feature representations.

[0038] A skip-modal reconstruction module is constructed to perform inverse decoding on the joint modal feature representation and output the reconstructed representation of the main modal features. Based on the unreachable regions marked by the unreachable region label in the main modal data, the main modal channel is skipped, the alternative modal path is activated, the reconstruction compensation operation is performed, and an alternative confidence score corresponding to the reconstructed representation is generated.

[0039] Optionally, step five specifically includes:

[0040] Set a preset confidence threshold, perform an item-by-item comparison operation on the substitution confidence scores output by the skip modality reconstruction module, and mark the substitution modality features with substitution confidence scores greater than the preset confidence threshold as valid substitution modality features;

[0041] Within the missing video image data region marked in the main modality missing identifier map, the effective alternative modality features at the corresponding time step are called as reconstruction input, the skip modality path is activated, the reconstruction compensation processing of the main modality features is performed, and the completed feature map patch within the main modality missing region is output.

[0042] Spatially fuse the completed feature map blocks with the original master modality feature map to generate an initial modality feature map;

[0043] Construct a main modality missing identifier map, set the pixels in the missing area to 1, and the pixels in the non-missing area to 0, to form a reconstructed region mask map;

[0044] Perform a pixel-by-pixel weighted fusion operation on the initial modal feature map and the reconstructed region mask map to fuse the reconstructed compensation region and the original preserved region, and output the main modal feature mask map; record the spatial location index information of the completed region and the preserved region in the main modal feature mask map to complete the generation of the main modal feature reconstruction map.

[0045] Optionally, step six specifically includes:

[0046] Construct a spatial weight adjustment matrix, assign a reduction weight coefficient to the position marked as the completion region in the main modality feature mask image, and assign a baseline weight coefficient to the position marked as the original retention region;

[0047] The spatial weight adjustment matrix and the corresponding original main modal features in the main modal feature mask are subjected to element-level weighting operations to obtain the original feature weighted map.

[0048] The spatial weight adjustment matrix and the corresponding completion features in the main modality feature mask map are subjected to element-level weighting operations to obtain the completion feature weighted map.

[0049] Perform a pixel-by-pixel fusion operation on the original feature weighted map and the completed feature weighted map in the spatial dimension to generate a fused feature map;

[0050] The fused feature map is expanded and compressed to extract multi-scale spatial structure features and temporal evolution features; the extracted spatial structure features and temporal evolution features are then concatenated into a fused feature vector in a unified format.

[0051] The fused feature vector is input into a fully connected network module, where feature mapping transformation is performed, and a disaster state vector with uniform dimensions is output.

[0052] Optionally, step seven specifically includes:

[0053] A time series buffer structure is constructed to store the disaster state vectors generated within a continuous acquisition period in an ordered manner according to time steps, forming a disaster state vector time series.

[0054] Perform Euclidean distance calculation on vector pairs between adjacent time steps in the disaster state vector time series to obtain the risk state change rate series;

[0055] Perform a sliding window mean smoothing operation on the risk state change rate sequence to generate a risk volatility gradient sequence; arrange the risk volatility gradient sequence in chronological order to construct a risk volatility trend curve;

[0056] Set a risk fluctuation threshold, compare the fluctuation gradient of the risk fluctuation trend curve over a continuous time period with the threshold, and identify the time period that meets the condition that the fluctuation gradient is continuously greater than the preset risk fluctuation threshold.

[0057] The identified high-risk fluctuation period and the corresponding disaster state vector at the time step are jointly input into the risk level classification module, the classification operation is performed, and the corresponding risk level identifier is output.

[0058] Based on the risk level identifier, the corresponding debris flow warning level is matched by looking up the table, and a debris flow warning instruction containing the warning level, duration and start time is output.

[0059] Optional, the AI-based radar-visual fusion intelligent monitoring and early warning device for debris flows includes the following modules:

[0060] Data acquisition module: used to acquire radar ranging data sequences, video image data sequences, hydrological observation data sequences and infrared image data sequences, and generate standardized multimodal monitoring data;

[0061] Modality detection module: used to construct modality data quality scoring functions, perform modality reachability detection, generate reachability label maps, and identify inaccessible areas of the main modality data;

[0062] Feature Reconstruction Module: Used to construct modality mapping maps and improved DANN models, extract candidate alternative features based on unreachable regions of the main modality data, and generate a main modality feature mask map;

[0063] Fusion Analysis Module: Used to construct a spatial weight adjustment mechanism, fuse the original main modal features and the completed features, generate fused feature maps and fused feature vectors, and extract disaster state vectors;

[0064] Early warning output module: used to construct risk fluctuation trend curves, combine disaster status vectors to perform risk level classification, and output debris flow early warning instructions.

[0065] The beneficial effects of this invention are:

[0066] This invention addresses the issue of video image data being susceptible to environmental interference and failure during debris flow monitoring by constructing a joint processing framework of reachability label maps and an improved DANN model. It introduces multimodal data from infrared thermal imagers, radar ranging, and hydrological observations as compensation sources. In the signal acquisition stage, time synchronization and spatial alignment strategies are employed to standardize multimodal monitoring data. During the modal diagnosis stage, modal reachability detection is performed based on a modal data quality scoring function to identify inaccessible areas of the primary modality and construct a set of alternative modal paths. In the feature reconstruction process, an improved DANN model is constructed, incorporating a multi-path modal encoder, a modal confidence estimator, and an adaptive adversarial module. This model dynamically completes missing features of the primary modality using a jump-modal reconstruction mechanism, generating a primary modality feature mask map and alternative confidence scores. In the feature fusion process, a disaster state vector is generated by constructing a spatial weight adjustment matrix and a weighted fusion strategy. Finally, in the trend identification stage, a risk fluctuation trend curve is constructed based on the disaster state vector, and dynamic level classification is performed, effectively improving the timeliness and accuracy of debris flow risk identification. Especially in scenarios with low signal-to-noise ratios or malfunctioning video images, such as at night, in dense fog, or during heavy rainfall, infrared image data can still respond stably, enabling reliable perception and early warning command output for high-risk areas. Ultimately, this achieves intelligent support for the entire process of debris flow risk status location, level determination, and command output. Attached Figure Description

[0067] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0068] Figure 1 The flowchart shows the intelligent monitoring and early warning method for debris flow based on the fusion of radar and vision proposed in this invention.

[0069] Figure 2 This is a structural diagram of the improved DANN model proposed in this invention;

[0070] Figure 3 This is a block diagram of the AI-based radar-visual fusion intelligent monitoring and early warning device for debris flows proposed in this invention. Detailed Implementation

[0071] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0072] refer to Figure 1-2 The method and equipment for intelligent monitoring and early warning of debris flows based on the integration of radar and vision, using artificial intelligence, include the following steps:

[0073] Step 1: Collect multimodal monitoring data within the debris flow monitoring area, including lidar point cloud data, visible light images, infrared images, and vibration signals collected by ground sensors. Then, synchronize and standardize the data of each modality according to time steps to construct a multimodal monitoring dataset.

[0074] Step 2: Set signal quality evaluation criteria based on signal-to-noise ratio, time synchronization rate, and modal integrity. Perform modal reachability detection on each modal data at each time step to obtain a modal reachability label map. By comparing the spatial distribution differences and noise disturbance intensity between the main mode and other modes, identify the signal-inaccessible regions in the main mode and mark the index positions of the inaccessible regions in both time and space dimensions.

[0075] Step 3: Construct a modality mapping map based on the physical acquisition path, spatial overlap area, and information redundancy of each modality; based on the inaccessible area of ​​the main modality in the label map, retrieve modality paths with similar spatial coverage and information content, and construct a set of alternative modality paths that can be used to compensate for the missing area of ​​the main modality; extract candidate alternative features for the corresponding time step and region from the set.

[0076] Step 4: Input the candidate alternative features into the improved DANN model, which introduces a modality confidence estimation branch and a jump feature decoupling structure; the model output includes the reconstructed representation of the main modality features and the alternative confidence score between each set of alternative features and the main modality.

[0077] Step 5: Compare the substitution confidence score with the preset confidence threshold. If the substitution confidence score is greater than the preset confidence threshold, perform a skip modality reconstruction operation in the corresponding unreachable region, use the reconstruction representation to complete the missing main modality features, and construct a main modality feature mask map to indicate whether each position is the original main modality feature or the completed feature.

[0078] Step 6: Based on the main modality feature mask, the original main modality features and the reconstructed and completed features are weighted and fused through a spatial weight adjustment mechanism. The fusion process considers the confidence level, perturbation intensity and semantic consistency of different regions to generate the fused disaster state vector.

[0079] Step 7: Based on the fused disaster state vector, construct a risk fluctuation trend curve in the time dimension; identify the time period when the risk fluctuation level is greater than the set fluctuation threshold, and determine the trigger window of the debris flow event; output debris flow early warning instructions in the corresponding time period, and record the spatial coordinates and characteristic evolution process of the trigger area.

[0080] This implementation constructs a modal reachability label map and identifies regions where the main modality is missing. It then uses a modal mapping map and an improved DANN model to perform skip-mode reconstruction, effectively achieving highly robust disaster state perception in scenarios where the main modality data is unavailable. By integrating original and completed features through a spatial weight adjustment mechanism, the stability and accuracy of the disaster state vector are improved. Finally, based on the risk fluctuation trend curve, a highly sensitive early warning of debris flows is achieved, possessing technical advantages such as strong adaptability, low false alarm rate, and high tolerance for data fragmentation.

[0081] In this embodiment, step one specifically includes:

[0082] In the key debris flow monitoring area, observation points are selected and radar ranging sensing modules are deployed. The radar ranging sensing modules are built based on pulse millimeter-wave radar devices, which collect information on changes in ground height and instantaneous distances to obstacles in the monitoring area, and generate radar ranging data sequences in time series form.

[0083] A high-definition visible light video image acquisition module is set up around the observation point. The video image acquisition module has a high frame rate automatic exposure adjustment function, acquires a continuous image frame sequence, and generates a standardized video image data sequence after performing compression encoding.

[0084] A hydrological information acquisition module is deployed along the debris flow channel. The hydrological information acquisition module includes a flow velocity monitoring unit, a flow meter and a sediment content measurement electrode to collect dynamic change data of flow velocity, flow rate and sediment content during the debris flow process and construct a hydrological observation data sequence.

[0085] An infrared thermal imaging module is deployed in the area overlapping with the field of view of the visible light image acquisition module. The infrared thermal imaging module is constructed by an infrared thermal imager based on a long-wave uncooled thermal imaging device. It acquires continuous thermal infrared image frames and generates an infrared image data sequence. The infrared image data sequence contains surface thermal radiation information and local temperature gradient distribution, identifies the temperature difference structure between water and solid particle components in debris flow, and enhances the image observation capability under low light and visual interference environments.

[0086] Time synchronization processing is performed on radar ranging data sequences, video image data sequences, hydrological observation data sequences, and infrared image data sequences respectively. The time synchronization processing is based on a unified timing signal to construct a synchronization timestamp mechanism and interpolates and aligns the data of each modality according to the acquisition cycle. Spatial alignment processing is performed on the observation data of each modality. The spatial alignment processing includes geographic coordinate mapping, viewpoint reconstruction, and pixel-level spatial index establishment to unify the spatial distribution reference system of each modality data.

[0087] By collecting, synchronizing, and aligning the original four-modal observation data, standardized multimodal monitoring data containing unified time and spatial indices is generated. This standardized multimodal monitoring data serves as the basic input for subsequent modal quality assessment, feature reconstruction, and disaster identification.

[0088] This implementation method enhances the perception blind spots of video image data by introducing infrared thermal imaging information, and still has stable temperature imaging capabilities under complex meteorological conditions such as night, dense fog, and heavy rainfall, effectively enhancing the monitoring system's all-time response capability to sudden disasters.

[0089] In this embodiment, step two specifically includes:

[0090] A modal data quality scoring function is constructed, comprising a signal-to-noise ratio (SNR) scoring function, a missing rate scoring function, and a modal continuity scoring function. The SNR scoring function evaluates the effective signal strength to noise ratio of each modality at each time step. It analyzes the local fluctuations of each modality's data through a preset sliding window and performs normalization to obtain the SNR score. The missing rate scoring function calculates the number of missing data points for each modality within a fixed time window and normalizes the calculation based on the global data volume, outputting the missing rate score. The modal continuity scoring function calculates the continuity score of the data change trend within adjacent time steps and outputs the continuity score. The weight parameters of the modal continuity scoring function are obtained by fitting a model based on a trade-off between modal robustness and early warning sensitivity.

[0091] The above three scores are input into the modal data quality scoring function to generate a quality score vector for each modality at each time step. The quality score vector is compared with the set signal quality evaluation criteria. The signal quality evaluation criteria are marked as reachable or inaccessible based on whether the signal-to-noise ratio scoring threshold, the missing rate scoring threshold, and the continuity scoring threshold are met, thus generating a modal reachability label set.

[0092] Perform a time-step expansion operation on the modal reachability label set to construct a reachability label graph. Each row in the reachability label graph represents the temporal reachability state of a modality, and each column represents the multimodal reachability combination state at the corresponding time step. Assign an independent label channel to each modality and use a binary mask to label the reachability results of each modality at each time step.

[0093] The video image data sequence is set as the main modality data. In the video image channel, consecutive time steps with a signal-to-noise ratio score value less than a preset signal-to-noise ratio threshold, a missing rate score value greater than a preset missing rate threshold, or a continuity score value less than a preset continuity threshold are selected as the main modality unreachable segments, and their start and end time indices and corresponding spatial regions are recorded. The above information is used to construct an unreachable region label map of the main modality data. Each pixel in the unreachable region label map contains the time index of the frame to which it belongs, the spatial location index, and the reachability status code.

[0094] Perform the fusion operation of the unreachable region label map and the reachability label map of the main modality data. Use logic and rules to obtain the main modality missing location and multimodal reachability status at each time step, and generate a multimodal fused label map containing spatial location mapping and modal channel labels.

[0095] In this embodiment, step three specifically includes:

[0096] Based on historical multimodal monitoring data, a modal mapping map is constructed. By setting a unified spatial reference coordinate system and a unified time step interval, modal node sets are constructed for radar ranging data sequences, video image data sequences, hydrological observation data sequences, and infrared image data sequences, respectively. On this basis, according to constraints such as spatial distance, time step consistency, and observation frequency, node matching operations are performed to construct a set of mapping edges between modal nodes, forming a mapping relationship structure between radar ranging channels, video image channels, hydrological observation channels, and infrared image channels. Using a joint probability estimation method, the feature distribution overlap degree is calculated between each pair of modal nodes with mapping edges. The overlap degree score is obtained by fitting a maximum likelihood estimation function to form a joint distribution similarity index matrix, which is used to characterize the semantic closeness between multimodal data at a specific spatiotemporal point.

[0097] The unreachable region label map of the main modality data is input into the path matching module. The corresponding unreachable region label map is composed of the main modality missing region encoding map output by the skip modality reachability analysis module. In the modality mapping map, all mapping paths containing the main modality node are traversed, and the overlap ratio between each mapping path and the main modality missing region in spatial coordinates is calculated. The set of paths with an overlap ratio greater than the spatial overlap threshold is selected to form a set of mapping paths aligned with the main modality missing region.

[0098] In the set of mapping paths aligned with the missing region of the main modality, the feature nodes of the source modality and the target modality connected by the path are read, and their corresponding joint distribution similarity scores are extracted. If the score is greater than the preset confidence threshold, the corresponding path is included in the candidate set of alternative modality paths. The confidence threshold is determined by the confidence interval model obtained by backfitting the out-of-sample multimodal prediction error.

[0099] Based on the alternative modal path candidate set, the node index positions of the radar ranging channel, hydrological observation channel and infrared image channel involved in the candidate path are determined, and the feature vectors corresponding to the time step and spatial position are extracted from the original radar ranging data sequence, hydrological observation data sequence and infrared image data sequence to form a candidate alternative feature set; the feature vectors include monitoring features such as distance change rate, average flow velocity, instantaneous flow rate and sediment concentration ratio.

[0100] In this embodiment, step four specifically includes:

[0101] A multi-channel modal encoder is constructed to receive candidate substitution features from three alternative modal channels; among them, the radar ranging channel is input to the radar coding submodule, the hydrological observation channel is input to the hydrological coding submodule, and the infrared image channel is input to the infrared coding submodule.

[0102] Each encoding submodule independently constructs a convolutional encoding network structure, performs feature extraction operations on the input features, and outputs the corresponding low-dimensional modality embedding representation. The low-dimensional modality embedding representation maintains a consistent length in the channel dimension and preserves structural heterogeneity in the modality dimension. Each convolutional encoding network includes multiple convolutional layers, normalization layers, and activation function structures, and extracts local spatial patterns through a max pooling structure. The weight parameters of the encoding network are trained independently to prevent cross-modal representation interference.

[0103] A modal confidence estimator is constructed by performing linear mapping and normalization operations on three low-dimensional modal embedding representations to generate three confidence vectors. The dimension of the confidence vectors is consistent with the number of channels in the embedding representations, and normalization is performed using the softmax function. Each confidence vector is compared with a preset confidence threshold to construct a modal dynamic weighting factor for each modality. The modal dynamic weighting factor includes a modal significance component and a modal stability component.

[0104] The modal saliency component is constructed based on the mean and variance of the activation values ​​of each channel in the low-dimensional modal embedding representation. The significance strength is jointly characterized by the channel response amplitude and the local feature distribution density. The saliency component is obtained by fitting a saliency scoring function. The modal stability component is constructed based on the range of confidence changes and volatility within the sliding time window. Specifically, it includes the difference between the maximum and minimum confidence values ​​and the standard deviation. The stability component is obtained by fitting a stability scoring function.

[0105] An adaptive adversarial module is constructed, which receives three modality embedding representations and corresponding modality dynamic weighting factors. The three low-dimensional modality embedding representations are concatenated according to the channel dimension and used as the fusion input. The modality dynamic weighting factors are used as the guiding signal to perform feature adversarial alignment operation. The adaptive adversarial module includes a gradient inversion structure, a modality domain discriminator and a shared coding structure. The modality domain discriminator enhances cross-modality distribution consistency through adversarial training and outputs joint modality feature representations.

[0106] A skip-modal reconstruction module is constructed to perform inverse decoding on the joint modal feature representation to generate a reconstructed representation of the main modality features. The inverse decoding structure is a symmetric convolutional decoding network, which includes an upsampling structure, a convolutional structure, and an activation layer structure. Based on the unreachable regions marked by the main modality data in the input, the main modality channel input is skipped, and the reconstruction channel constructed by the alternative modality path is activated to perform a skip-reconstruction compensation operation. After outputting the main modality reconstructed feature map, the corresponding alternative confidence score is generated by combining the reconstruction error and the discrimination confidence.

[0107] This implementation improves the alternative reconstruction capability in complex environments where the main mode is missing by independently encoding and jointly adversarially aligning the three heterogeneous modes. The infrared image mode can still stably provide thermal radiation information in scenarios where the signal-to-noise ratio of visible light video decreases, such as at night, in dense fog, and in heavy rain, thereby enhancing the environmental robustness and reconstruction stability of the overall model.

[0108] In this embodiment, the improved DANN model is specifically as follows:

[0109] A multi-modal encoder structure is constructed. The multi-modal encoder receives candidate alternative features from three alternative modal channels and performs convolutional transformation, feature compression, and embedding mapping operations through three independent convolutional coding networks to generate low-dimensional modal embedding representations of the three modalities. The three convolutional coding networks have the same network depth and activation structure in structure, but remain completely independent in parameter space to enhance the ability to model modal differences.

[0110] A modal confidence estimator module is constructed. The modal confidence estimator receives low-dimensional modal embedding representations of three modalities, performs linear transformation mapping operations and normalization processing on each, and obtains three confidence vectors. The confidence vectors are compared with the set confidence thresholds, and dynamic weighting factors for the three modalities are generated based on the comparison results. The dynamic weighting factors are composed of modal saliency components and modal stability components.

[0111] Modal saliency components are constructed. These components are obtained by jointly modeling the feature mean and variance of the modal embedding representation in the channel dimension, extracting the response amplitude and distribution density features of the modality in the spatial dimension. A saliency scoring function is used to generate a modal saliency score by fitting the weighted relationship between the average activation value and the activation distribution variance of each modal channel, thus obtaining the modal saliency components. The saliency scoring function is obtained by fitting the average activation value and the activation distribution variance.

[0112] Modal stability components are constructed; these components are modeled based on the confidence range and confidence volatility of the corresponding modes within a historical time window; for each confidence vector, the difference between its maximum and minimum values ​​and its standard deviation are calculated within a set sliding time window, and the calculation results are input into a stability scoring function to generate a stability score, thereby obtaining the modal stability components; the stability scoring function is obtained by weighted fitting of the confidence range and confidence volatility.

[0113] An adaptive adversarial module is constructed. The adaptive adversarial module receives three modality embedding representations and corresponding modality dynamic weighting factors, and performs modality alignment adjustment operations. By multiplying each modality channel by the corresponding dynamic weighting factor and inputting it into a shared structure modality domain discriminant network, modality distribution adversarial training is performed using a gradient inversion structure. During the adversarial training process, the consistency of the distribution of the source modality and the target modality is used as the optimization objective to guide the modality domain discriminant network to output joint modality feature representations.

[0114] A skip modality reconstruction module is constructed. The skip modality reconstruction module receives the joint modality feature representation, performs inverse decoding to recover the embedded feature distribution of the main modality, automatically masks the main modality channel path and activates the alternative modality path based on the unreachable region label in the input, and performs reconstruction compensation operation based on the joint modality features. The alternative representation of the main modality is generated by decoding and reconstructing the alternative modality features, and the similarity between the alternative representation and the real main modality is calculated, and the alternative confidence score is output.

[0115] The modal confidence estimator, adaptive adversarial module, and skip modal reconstruction module jointly constitute a modal adaptive processing path, which can achieve robust feature compensation and discrimination enhancement under the condition that the main modality is missing or the noise of the substitute modality is disturbed. Through the synergistic effect of the above structure and processing steps, the technical effect of maintaining the overall system discrimination performance is achieved in scenarios with incomplete multimodal inputs or inconsistent signal and noise.

[0116] In this embodiment, step five specifically includes:

[0117] Set a preset confidence threshold, perform an item-by-item comparison operation on the alternative confidence scores output by the skip modality reconstruction module, and filter out alternative modality features with confidence scores greater than the preset confidence threshold, and mark them as valid alternative modality features;

[0118] Construct a main modality missing marker map, marking the pixel positions within the main modality missing region as 1 and other regions as 0;

[0119] Based on the missing areas of video image data marked in the main modality missing identifier map, the spatial index information of the missing areas in the video image channels of the corresponding time steps is extracted as a spatial constraint for the reconstruction target area.

[0120] For each missing region, the effective alternative modal features of the corresponding time step are called and input into the skip modal path, skipping the main modal channel, performing reconstruction compensation processing, and outputting the complete feature map of the missing region of the main modality;

[0121] The completed feature map blocks are inserted into the missing positions of the original master modality feature map to form an initial modality feature map containing the completed content;

[0122] A reconstruction region mask map is constructed based on the main modality missing identifier map. The missing regions are assigned reconstruction weights α, and the retained regions are assigned values ​​of 1, forming a weight mask matrix for fusion, where α is a fusion reduction coefficient less than 1.

[0123] The initial modal feature map and the reconstructed region mask map are subjected to a pixel-by-pixel weighted fusion operation. The missing regions retain the complete features, and the non-missing regions retain the original master modal features, and the master modal feature mask map is output. The spatial location index information of the completed region and the retained region in the master modal feature mask map is recorded, which is used for the subsequent fusion analysis module to process regional differences when generating disaster state vectors.

[0124] This implementation method controls the activation conditions of skip modal paths by using confidence screening, accurately locates missing regions by combining the main modality missing identifier map, and achieves spatial fusion control by reconstructing the region mask map. This effectively improves the accuracy and reliability of main modality feature completion and can maintain the stability and continuity of disaster state estimation even when the main modality data is severely missing.

[0125] In this embodiment, step six specifically includes:

[0126] Construct a spatial weight adjustment matrix, extract the spatial position indices of the completed region and the original preserved region in the main modality feature mask map, assign the position of the completed region to the reduction weight coefficient β according to the set strategy, assign the position of the original preserved region to the benchmark weight coefficient 1, and form a spatial fusion weight map with the same dimension as the main modality feature map.

[0127] Extract the original main modality feature map and the completed feature map corresponding to the main modality feature mask map, and perform element-level pixel-wise multiplication operations with the spatial fusion weight map in the spatial dimension to obtain the original feature weight map and the completed feature weight map.

[0128] The original weighted feature map and the completed weighted feature map are fused pixel by pixel according to spatial coordinates, and the fused feature map with smooth spatial transition is output.

[0129] Multi-scale channel unfolding operation is performed on the fused feature map, and different convolution kernel sizes are introduced to extract local and global spatial structure features respectively. Then, a unified dimension mapping is performed through the channel compression module to construct a multi-scale spatial feature tensor.

[0130] The fused feature maps are stacked in chronological order to form a fused feature time series tensor, and a one-dimensional convolution operation is performed on the time axis to extract the time evolution feature tensor.

[0131] The multi-scale spatial feature tensor and the temporal evolution feature tensor are concatenated in the feature dimension direction to form a fused feature vector;

[0132] The fused feature vector is input into a fully connected network module, and linear mapping, activation function and dimensionality compression operations are performed in sequence to output a disaster state vector with fixed dimensions.

[0133] This implementation method constructs a spatial fusion weighting mechanism to precisely control the feature contribution ratio during the original and complete processes; combined with multi-scale spatial structure modeling and temporal dynamic modeling capabilities, it effectively enhances the response capability of the disaster state vector to geomorphological changes and temporal evolution trends, thereby improving the sensitivity and discrimination accuracy of debris flow early warning.

[0134] In this embodiment, step seven specifically includes:

[0135] A time series buffer structure is constructed, and a sampling window of fixed length is set. The disaster state vectors generated within the continuous acquisition period are sequentially stored into the buffer structure according to the time step order to form a disaster state vector time series. Euclidean distance is calculated for any two adjacent time step disaster state vectors in the disaster state vector time series to construct a risk state change rate sequence. A sliding window length is set for the risk state change rate sequence, and sliding window mean smoothing is performed to generate a risk fluctuation gradient sequence. The risk fluctuation gradient sequence is arranged according to the time step to construct a risk fluctuation trend curve, and a mapping relationship between the time axis and the risk fluctuation gradient value is established.

[0136] A risk fluctuation threshold is set, and all time points in the risk fluctuation trend curve are traversed to identify the set of time periods where the fluctuation gradient value is greater than the risk fluctuation threshold within a continuous time period. The start time and duration of each time interval are recorded. The disaster state vector within the high-risk fluctuation time period is extracted and combined with the corresponding timestamp to construct a risk state segment set. The risk state segment set is input into the risk level classification module, which contains multiple preset risk level template features. By calculating the similarity score between the input state vector and the level template and performing the maximum match, the corresponding risk level identifier is output.

[0137] Establish a risk level mapping table, with each item including a risk level identifier, corresponding warning level, trigger threshold, minimum duration, and response strategy recommendations. Based on the risk level identifier, look up the corresponding debris flow warning level information in the table to generate a debris flow warning command containing the warning level, risk duration, and start time. Synchronize the warning command to the warning output module to drive the audible and visual alarm devices and the remote communication module to perform warning push operations.

[0138] This step introduces continuous state change rate modeling and risk level matching mechanism to identify abrupt change signals in the dynamic change trend of disaster state vector, which can detect and accurately classify potential debris flow risks in advance, significantly improving the response speed and risk classification accuracy of the early warning system in sudden disaster scenarios.

[0139] refer to Figure 3 The AI-based radar-visual fusion intelligent monitoring and early warning device for debris flows includes the following modules:

[0140] Data acquisition module: used to acquire radar ranging data sequences, video image data sequences, hydrological observation data sequences and infrared image data sequences, and generate standardized multimodal monitoring data;

[0141] Modality detection module: used to construct modality data quality scoring functions, perform modality reachability detection, generate reachability label maps, and identify inaccessible areas of the main modality data;

[0142] Feature Reconstruction Module: Used to construct modality mapping maps and improved DANN models, extract candidate alternative features based on unreachable regions of the main modality data, and generate a main modality feature mask map;

[0143] Fusion Analysis Module: Used to construct a spatial weight adjustment mechanism, fuse the original main modal features and the completed features, generate fused feature maps and fused feature vectors, and extract disaster state vectors;

[0144] Early warning output module: used to construct risk fluctuation trend curves, combine disaster status vectors to perform risk level classification, and output debris flow early warning instructions.

[0145] Example 1:

[0146] To verify the feasibility of this invention in practice, it was applied to a typical high-risk debris flow mountainous area. This region has a large topographical drop, concentrated rainfall, and narrow valleys, possessing typical disaster triggering conditions. The system selected high-position radar and infrared thermal imaging acquisition equipment on both sides of the gully mouth, deployed hydrological observation nodes in the middle section, and installed video image acquisition modules on the monitoring towers and protective platforms, ultimately constructing a complete radar-video-infrared-hydrological fusion sensing network.

[0147] During the three-month continuous monitoring period, a total of 26,870 image frames, 41,000 sets of radar ranging data, 19,200 hydrological observation data, and approximately 11,000 infrared image frames were acquired. Heavy rainfall was recorded for 17 days during the same period, with a cumulative period of 29 hours of severe radar signal interference. 573 video modalities were lost due to dense fog or insufficient nighttime lighting. By setting a modal data quality scoring function, signal-to-noise ratio analysis, modal continuity detection, and missing rate statistics were performed on the multimodal data to generate an accessibility label map and identify inaccessible areas in the video modalities.

[0148] When dealing with inaccessible areas, an improved DANN model is invoked, inputting the embedded features of three alternative modalities and performing a skip-based master modality reconstruction. Simultaneously, confidence scores are fused to output the final reconstructed image and feature mask map. Infrared thermal imaging data demonstrates significant advantages at night, maintaining temperature difference identification of water bodies and rocks even in complete darkness, providing crucial support for master modality completion. Based on this, the system combines all modalities to construct a disaster state vector and establish a risk fluctuation trend curve, triggering disaster early warning commands and recording early warning response status.

[0149] Table 1. Comparison of Modal Missing Entries and Recognition Accuracy

[0150] project Traditional methods Method of the present invention Total number of missing frames for a modality (main modality data) 573 573 Completely lost frame recovery number 78 412 Some features can be successfully reconstructed from frames. 123 149 Mean error of modal reconstruction (MSE) 0.114 0.036 Alternate mode confidence average none 0.853 The accuracy of recognition is improved (%) after modality fusion. none +27.4%

[0151] Table 1 illustrates the completion capability of the method of this invention in scenarios where the main modality data is missing. Traditional methods can only recover 78 frames through historical mean estimation, and the reconstruction results are blurry and feature incomplete. In contrast, this invention utilizes a multimodal embedding feature combined with an improved DANN model for a skip-reconstruction approach, successfully recovering 412 frames out of 573 missing frames, with the average reconstruction error significantly reduced to 0.036. Simultaneously, the average confidence score of the alternative modality is as high as 0.853, indicating that the alternative channel possesses stability and representativeness. The final recognition accuracy is improved by 27.4%, significantly outperforming traditional methods, fully demonstrating the adaptability and anti-interference performance of this embodiment in the case of lost disaster image data.

[0152] Table 2. Evaluation Table of Accuracy and Effectiveness of Risk Fluctuation Detection and Early Warning

[0153] Time period Actual mudslide events System early warning output Early response time (min) Was the early warning successful? Disaster Level Determination 1 yes yes 22 success medium 2 no yes — False alarm Low 3 yes yes 15 success high 4 yes yes 33 success high 5 no yes — False alarm medium 6 yes yes 19 success medium 7 yes yes 11 success high

[0154] Table 2 presents the analysis of the effectiveness of the early warning system during three months of continuous operation. A total of seven high-risk fluctuations were identified, five of which closely matched actual debris flow events, resulting in a successful early warning rate of 71.4%. The response time for each successful early warning ranged from 11 to 33 minutes, fully meeting emergency response requirements. Two false alarms were caused by strong vibrations or terrain disturbances, corresponding to low or moderate disaster levels, and did not trigger actual alarm operations. This embodiment demonstrates superior performance in terms of continuity, advance warning, and robustness. It can operate stably even in complex environments such as nighttime and dense fog conditions, relying on infrared thermal imaging modes, ensuring all-weather, all-time disaster perception capabilities.

[0155] This embodiment comprehensively verifies the practicality and cutting-edge nature of the invention in typical debris flow disaster scenarios. It not only improves the accuracy of missing data completion but also significantly enhances the timeliness and reliability of intelligent early warning, possessing broad engineering application value. The system as a whole demonstrates strong robustness in disaster identification under extreme environments, meeting the technological development trends of modern geological disaster monitoring equipment towards intelligence, precision, and all-scenario adaptability.

[0156] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An AI-based intelligent monitoring and early warning method for debris flows using a fusion of radar and visual imaging, characterized in that: Includes the following steps: Step 1: Collect multimodal monitoring data within the debris flow monitoring area; Step 2: Based on the established signal quality assessment criteria, perform modal reachability detection on the multimodal monitoring data to generate an reachability label map; set the video image data sequence in the multimodal monitoring data as the primary modal data, and mark the inaccessible areas of the primary modal data in the reachability label map; Step 3: Construct a modality mapping map, determine the set of alternative modal paths based on the inaccessible regions, and extract the corresponding candidate alternative features; Step 4: Input the candidate alternative features into the improved DANN model, and output the reconstructed representation of the main modality features and the alternative confidence score; Step 5: Based on the alternative confidence score and the preset confidence threshold, perform a skip modality reconstruction operation to fill in the missing main modality features in the inaccessible region and generate a main modality feature mask map; Step 6: Combining the aforementioned master modality feature mask, a spatial weight adjustment mechanism is used to fuse the original master modality features and the completed features to generate a disaster state vector; Step 7: Construct a risk fluctuation trend curve based on the disaster state vector, identify the time period when the risk fluctuation level is greater than the preset fluctuation threshold, and generate the corresponding debris flow early warning instruction.

2. The intelligent monitoring and early warning method for debris flow based on artificial intelligence and fusion of radar and vision as described in claim 1, characterized in that, Step one specifically involves: A radar ranging sensor module is deployed within the target area to collect data on changes in ground elevation and distances to obstacles, generating a radar ranging data sequence; a video image acquisition module is deployed to collect a continuous sequence of image frames, generating a video image data sequence; a hydrological information acquisition module is deployed to collect and calculate data on debris flow velocity, flow rate, and sediment content, generating a hydrological observation data sequence; and an infrared thermal imaging acquisition module is deployed to use a long-wave uncooled thermal imaging device in an infrared thermal imager to collect a sequence of surface temperature distribution images, identify the temperature difference distribution between water and solid particles in the debris flow, and generate an infrared image data sequence. Time synchronization and spatial alignment processing are performed on radar ranging data sequences, video image data sequences, hydrological observation data sequences, and infrared image data sequences to unify the timestamps and spatial indexes of each modality of data and generate standardized multimodal monitoring data.

3. The AI-based radar-visual fusion intelligent monitoring and early warning method for debris flows according to claim 2, characterized in that, Step two specifically involves: A modal data quality scoring function is constructed to calculate the signal-to-noise ratio, data missing rate, and modal continuity score for radar ranging data sequences, video image data sequences, hydrological observation data sequences, and infrared image data sequences, respectively. Based on the established signal quality assessment criteria and the output of the modal data quality scoring function, the reachability status of each mode at each time step is marked. Arrange the reachability states by time steps to generate an reachability label map, and assign a corresponding label channel to each mode; Based on the reachability label map, regions with signal-to-noise ratios less than a preset threshold or missing data within a continuous time step are marked in the video image channel, generating a main modality data unreachable region label map.

4. The intelligent monitoring and early warning method for debris flow based on radar-visual fusion according to claim 3, characterized in that, Step three specifically involves: A modal mapping map is constructed based on historical multimodal monitoring data. The modal mapping map includes the mapping relationship structure and joint distribution similarity index between radar ranging channels, video image channels, hydrological observation channels and infrared image channels. The mapping relationship structure establishes a set of mapping paths between modes based on space-time indexing rules. The joint distribution similarity index forms a similarity scoring matrix by calculating the overlap of feature distributions of each modal channel at the same spatial location and time step. Using the unreachable region label map of the main modality data as input, the set of mapping paths whose spatial locations coincide with the missing regions of the main modality are retrieved in the modality mapping map; Alternative modal paths with confidence scores greater than a preset confidence threshold are selected from the mapping path set to form an alternative modal path candidate set; Based on the alternative modal path candidate set, the corresponding reachable modal data index is determined, and feature vectors that satisfy the index constraints are extracted from the radar ranging data sequence, hydrological observation data sequence and infrared image data sequence to form candidate alternative features.

5. The AI-based intelligent monitoring and early warning method for debris flow fusion based on radar and vision as described in claim 4, characterized in that, Step four specifically involves: An improved DANN model is constructed; the improved DANN model includes a multi-channel modal encoder, a modal confidence estimator, an adaptive adversarial module, and a skip modal reconstruction module; The candidate substitution features are input into the multi-modal encoder, and low-dimensional feature representations are extracted from different modal channels respectively; The low-dimensional feature representation is input into the modality confidence estimator to generate confidence scores for each alternative modality; The low-dimensional feature representation and the confidence score are input into the adaptive adversarial module to perform feature domain alignment processing and generate aligned modality joint features. The modal joint features are input into the skip modality reconstruction module, which outputs the reconstructed representation of the main modality features and generates an alternative confidence score corresponding to the reconstructed representation.

6. The AI-based intelligent monitoring and early warning method for debris flow fusion based on radar and vision as described in claim 5, characterized in that, The improved DANN model is specifically as follows: A multi-channel modal encoder is constructed to receive candidate substitution features from three alternative modal channels, and low-dimensional modal embedding representations of the three alternative modalities are extracted through three independent convolutional coding networks. A modal confidence estimator is constructed by performing linear mapping and normalization on the low-dimensional modal embedding representation to generate three confidence vectors. Based on each confidence vector and a preset confidence threshold, a modal dynamic weighting factor is constructed. The modal dynamic weighting factor includes a modal significance component and a modal stability component. The modal saliency components are constructed based on the feature mean and variance in the modal embedding representation. The saliency of each modality is jointly characterized by the feature response amplitude and the feature space distribution density. The corresponding modal saliency components are generated by calculating the average activation value of each channel in the embedding space and the variance in the embedding dimension, and combining them with the saliency scoring function. The modal stability component is constructed based on the confidence variation range and volatility of the mode within a historical time window, and the modal stability is quantified by the maximum confidence difference and standard deviation within the sliding window. The modality stability components are generated by calculating the difference between the maximum and minimum values ​​and the standard deviation of the confidence vector of the modality within the time window and inputting them into the stability scoring function. An adaptive adversarial module is constructed, which receives three modality embedding representations and corresponding modality dynamic weighting factors, performs cross-modality feature distribution alignment operation, optimizes the distribution consistency between the source modality and the target modality through a gradient inversion structure and a modality domain discriminant network, and generates joint modality feature representations. A skip modality reconstruction module is constructed to perform inverse decoding on the joint modality feature representation and output the reconstructed representation of the main modality feature. If there is a main modality missing identifier in the input, the main modality channel is skipped, the alternative modality path is activated, the reconstruction compensation operation is performed, and an alternative confidence score corresponding to the reconstructed representation is generated.

7. The AI-based intelligent monitoring and early warning method for debris flow fusion based on radar and vision as described in claim 6, characterized in that, Step five specifically involves: Set a preset confidence threshold, perform an item-by-item comparison operation on the substitution confidence scores output by the skip modality reconstruction module, and mark the substitution modality features with substitution confidence scores greater than the preset confidence threshold as valid substitution modality features; Within the missing video image data region marked in the main modality missing identifier map, the effective alternative modality features at the corresponding time step are called as reconstruction input, the skip modality path is activated, the reconstruction compensation processing of the main modality features is performed, and the completed feature map patch within the main modality missing region is output. Spatially fuse the completed feature map blocks with the original master modality feature map to generate an initial modality feature map; Construct a main modality missing identifier map, set the pixels in the missing area to 1, and the pixels in the non-missing area to 0, to form a reconstructed region mask map; Perform a pixel-by-pixel weighted fusion operation on the initial modal feature map and the reconstructed region mask map to fuse the reconstructed compensation region and the original preserved region, and output the main modal feature mask map; Record the spatial location index information of the completed region and the preserved region in the master modality feature mask map to complete the generation of the master modality feature reconstruction map.

8. The intelligent monitoring and early warning method for debris flow based on artificial intelligence and fusion of radar and vision as described in claim 7, characterized in that, Step six specifically involves: Construct a spatial weight adjustment matrix, assign a reduction weight coefficient to the position marked as the completion region in the main modality feature mask image, and assign a baseline weight coefficient to the position marked as the original retention region; The spatial weight adjustment matrix and the corresponding original main modal features in the main modal feature mask are subjected to element-level weighting operations to obtain the original feature weighted map. The spatial weight adjustment matrix and the corresponding completion features in the main modality feature mask map are subjected to element-level weighting operations to obtain the completion feature weighted map. Perform a pixel-by-pixel fusion operation on the original feature weighted map and the completed feature weighted map in the spatial dimension to generate a fused feature map; The fused feature map is expanded and compressed to extract multi-scale spatial structure features and temporal evolution features. The extracted spatial structure features and temporal evolution features are concatenated into a unified fusion feature vector; The fused feature vector is input into a fully connected network module, where feature mapping transformation is performed, and a disaster state vector with uniform dimensions is output.

9. The intelligent monitoring and early warning method for debris flow based on radar-visual fusion according to claim 8, characterized in that, Step seven specifically involves: A time series buffer structure is constructed to store the disaster state vectors generated within a continuous acquisition period in an ordered manner according to time steps, forming a disaster state vector time series. Perform Euclidean distance calculation on vector pairs between adjacent time steps in the disaster state vector time series to obtain the risk state change rate series; Perform a sliding window mean smoothing operation on the risk state change rate sequence to generate a risk volatility gradient sequence; Arrange the risk volatility gradient sequence in chronological order to construct a risk volatility trend curve; Set a risk fluctuation threshold, compare the fluctuation gradient of the risk fluctuation trend curve over a continuous time period with the threshold, and identify the time period that meets the condition that the fluctuation gradient is continuously greater than the preset risk fluctuation threshold. The identified high-risk fluctuation period and the corresponding disaster state vector at the time step are jointly input into the risk level classification module, the classification operation is performed, and the corresponding risk level identifier is output. Based on the risk level identifier, the corresponding debris flow warning level is matched by looking up the table, and a debris flow warning instruction containing the warning level, duration and start time is output.

10. An AI-based radar-visual fusion intelligent monitoring and early warning device for debris flows, comprising the AI-based radar-visual fusion intelligent monitoring and early warning method for debris flows as described in any one of claims 1 to 9, characterized in that, Includes the following modules: Data acquisition module: used to acquire radar ranging data sequences, video image data sequences, hydrological observation data sequences and infrared image data sequences, and generate standardized multimodal monitoring data; Modality detection module: used to construct modality data quality scoring functions, perform modality reachability detection, generate reachability label maps, and identify inaccessible areas of the main modality data; Feature Reconstruction Module: Used to construct modality mapping maps and improved DANN models, extract candidate alternative features based on unreachable regions of the main modality data, and generate a main modality feature mask map; Fusion Analysis Module: Used to construct a spatial weight adjustment mechanism, fuse the original main modal features and the completed features, generate fused feature maps and fused feature vectors, and extract disaster state vectors; Early warning output module: used to construct risk fluctuation trend curves, combine disaster status vectors to perform risk level classification, and output debris flow early warning instructions.

Images