Deep learning-based disaster multi-source data real-time fusion early warning system

Abstract

The invention discloses a disaster multi-source data real-time fusion early warning system based on deep learning, which belongs to the technical field of disaster early warning and comprises a multi-source data space-time alignment module, a cross-modal semantic fusion module, a disaster evolution dynamic prediction module and a minute-level early warning generation module. According to the method, accurate alignment of multi-source data is realized through an adaptive space-time resampling algorithm, cross-modal deep semantic fusion is realized through a cross attention mechanism, a disaster evolution trend is predicted through a space-time diagram attention network, minute-level early warning response is realized through an incremental calculation strategy, and disaster early warning is realized. According to the method, precise space-time alignment of the multi-source heterogeneous data is realized through adaptive space-time resampling and timestamp synchronous correction, and the space-time alignment precision is improved.

Description

Technical Field

[0001] This invention relates to the field of disaster early warning technology, specifically to a real-time fusion early warning system for multi-source disaster data based on deep learning. Background Technology

[0002] Disaster early warning is a crucial link in the disaster prevention, mitigation, and relief system. Timely and accurate early warning information provides vital support for emergency response and proactive prevention decisions. With the rapid development of satellite remote sensing technology, IoT sensor networks, and social media platforms, the sources of disaster monitoring data are becoming increasingly diverse. Satellite remote sensing can provide large-scale information on land cover and vegetation changes, IoT sensors can collect real-time environmental parameters such as meteorological, hydrological, and geological data, and user-posted content on social media platforms contains a wealth of firsthand descriptions of disaster situations and on-site images. How to effectively integrate these multi-source, heterogeneous data and fully explore the complementary information between them has become a core technical challenge in improving disaster early warning capabilities.

[0003] Chinese invention CN118278577A discloses a BeiDou intelligent emergency rescue command and dispatch system. This system includes a positioning module for real-time communication with the BeiDou satellite navigation system, a data acquisition module, a big data processing and analysis module, a rescue resource database, an intelligent decision support module, an interactive command platform, and a system interface module. The data acquisition module includes a multi-source heterogeneous data fusion submodule, used to integrate disaster site data from various sensors, multispectral imaging equipment on UAVs, satellite remote sensing imagery, and social media sources. It achieves the fusion and management of multi-source heterogeneous data through data cleaning, format conversion, and spatiotemporal calibration. The big data processing and analysis module includes a deep learning prediction submodule, used to train historical disaster data, meteorological data, and geographic information data to construct a disaster development trend prediction model, predicting the disaster's spread, intensity changes, and secondary disaster indicators.

[0004] However, the technical solution in CN118278577A has the following technical problems. First, the multi-source heterogeneous data fusion submodule only performs routine preprocessing operations such as data cleaning, format conversion, and spatiotemporal calibration, lacking the ability to perform deep semantic fusion of heterogeneous data such as satellite remote sensing image data, IoT sensor data, and social media text and images. Data from different modalities have fundamentally different feature representation spaces. Simple format conversion and spatiotemporal calibration cannot effectively establish cross-modal semantic relationships, resulting in the fused features failing to fully reflect the complementary information between multi-source data. Second, the spatiotemporal calibration process only considers the unified transformation of coordinate systems, failing to effectively address the differences in sampling frequency and time delay between different data sources. Satellite remote sensing image data typically has hourly or dayly temporal resolutions, IoT sensor data has second- or minute-level sampling frequencies, and social media data has random and delayed posting times. These differences in time scales lead to the inability to accurately align multi-source data in the time dimension. Third, while conventional deep learning models such as Long Short-Term Memory (LSTM) networks, gated recurrent units (GRUs), and convolutional neural networks are used for disaster prediction, these models, although capable of processing single-modality time-series or image data, lack the ability to collaboratively model multi-source data and cannot effectively capture the interaction relationships and complementary features between different data sources. Fourth, the minute-level real-time early warning generation mechanism is not publicly disclosed. When disasters evolve rapidly, traditional batch processing methods are insufficient to meet the demand for timely early warnings. Fifth, the system architecture is a unidirectional data flow structure, and early warning results cannot be fed back to the data fusion and prediction modules to dynamically optimize system parameters. When external environmental changes or data characteristics drift, the system struggles to adapt and adjust.

[0005] Therefore, a disaster early warning system is needed that can achieve deep semantic fusion of multi-source heterogeneous data, accurate spatiotemporal alignment, collaborative predictive modeling, minute-level real-time early warning generation, and closed-loop feedback optimization, in order to solve the aforementioned technical problems in the existing technology. Summary of the Invention

[0006] The purpose of this invention is to provide a real-time disaster multi-source data fusion early warning system based on deep learning, in order to solve the technical problems in the prior art such as insufficient fusion depth of multi-source heterogeneous data, limited spatiotemporal alignment accuracy, lack of collaborative prediction modeling, insufficient early warning timeliness, and lack of closed-loop feedback mechanism.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] The disaster multi-source data real-time fusion early warning system based on deep learning includes a multi-source data spatiotemporal alignment module, a cross-modal semantic fusion module, a disaster evolution dynamic prediction module, a minute-level early warning generation module, and a closed-loop feedback optimization module.

[0009] The multi-source data spatiotemporal alignment module receives satellite remote sensing imagery, IoT sensor data, and social media data. It automatically adjusts resampling parameters based on the original sampling frequency and spatial resolution of each data source using an adaptive spatiotemporal resampling unit. A coordinate unification mapping unit maps spatial data from different coordinate systems to a unified geographic coordinate system. A timestamp synchronization correction unit eliminates time delay differences between different data sources. Finally, the module outputs spatiotemporally aligned multi-source data.

[0010] The cross-modal semantic fusion module receives spatiotemporally aligned multi-source data. It extracts visual feature vectors from satellite remote sensing imagery data using a remote sensing feature encoder. It also extracts temporal feature vectors from IoT sensor data using a sensor feature encoder. Furthermore, it extracts semantic feature vectors from social media data using a social semantic encoder. Finally, it calculates the correlation weights between feature vectors from different modalities using a cross-attention fusion unit based on a cross-attention mechanism and performs weighted fusion. The module outputs cross-modal fused features.

[0011] The disaster evolution dynamic prediction module receives cross-modal fused features. It constructs these fused features into a spatiotemporal graph structure. The module then uses a spatiotemporal graph attention network to perform message passing and aggregation updates on the node features within the spatiotemporal graph structure. Finally, the module predicts the disaster's spread, intensity changes, and evolution sequence. The module outputs the disaster evolution prediction results and prediction confidence scores.

[0012] The minute-level early warning generation module receives disaster evolution prediction results. It uses an incremental early warning generator to locally update new data based on an incremental calculation strategy. Within the early warning response time threshold, the module generates minute-level early warning information containing disaster type, affected area, warning level, and response recommendations.

[0013] Preferably, the resampling parameters of the adaptive spatiotemporal resampling unit include the temporal resampling interval and the spatial resampling resolution. The temporal resampling interval ranges from 1 minute to 60 minutes, and the spatial resampling resolution ranges from 10 meters to 1000 meters.

[0014] Preferably, the relevance weights in the cross-attention fusion unit are obtained by the dot product attention operation of the query vector, key vector, and value vector, and the relevance weights range from 0 to 1.

[0015] Preferably, the number of message passing rounds in the spatiotemporal graph attention network ranges from 2 to 6 rounds, and the prediction confidence score ranges from 0 to 1.

[0016] Preferably, the warning response time threshold of the incremental warning generator is in the range of 1 minute to 10 minutes.

[0017] Preferably, the warning levels for minute-level early warning information include four levels: blue warning, yellow warning, orange warning, and red warning.

[0018] The beneficial effects of this invention are as follows: it achieves accurate spatiotemporal alignment of multi-source heterogeneous data through an adaptive spatiotemporal resampling algorithm and a timestamp synchronization correction algorithm, improving the spatiotemporal alignment accuracy by about 40% compared with conventional calibration methods; it achieves cross-modal deep semantic fusion through a cross-attention fusion mechanism, improving the feature fusion completeness by about 35% compared with simple splicing methods; it models the spatiotemporal evolution of disasters through a spatiotemporal graph attention network, improving the prediction accuracy by about 25% compared with single-model methods; and it achieves minute-level early warning response through an incremental calculation strategy, controlling the early warning generation time to within 10 minutes. Attached Figure Description

[0019] Figure 1 This is the overall architecture diagram of the real-time fusion early warning system for disaster multi-source data based on deep learning, which is based on the present invention.

[0020] Figure 2 This is a detailed structural diagram of the multi-source data spatiotemporal alignment module of the present invention.

[0021] Figure 3 This is a detailed structural diagram of the cross-modal semantic fusion module of the present invention.

[0022] Figure 4 This is a detailed structural diagram of the disaster evolution dynamic prediction module of the present invention.

[0023] Figure 5 This is a detailed structural diagram of the minute-level early warning generation module and the closed-loop feedback optimization module of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0025] This invention provides a real-time disaster multi-source data fusion early warning system based on deep learning, such as... Figure 1As shown, the system comprises five core functional modules: a multi-source data spatiotemporal alignment module, a cross-modal semantic fusion module, a disaster evolution dynamic prediction module, a minute-level early warning generation module, and a closed-loop feedback optimization module. These five modules form a deeply coupled closed-loop collaborative architecture: the output of the multi-source data spatiotemporal alignment module serves as the input to the cross-modal semantic fusion module, the output of the cross-modal semantic fusion module serves as the input to the disaster evolution dynamic prediction module, and the output of the disaster evolution dynamic prediction module serves as the input to the minute-level early warning generation module. Simultaneously, the prediction confidence score output by the disaster evolution dynamic prediction module is transmitted to the closed-loop feedback optimization module. The closed-loop feedback optimization module generates parameter adjustment signals based on the confidence assessment results and feeds them back to the multi-source data spatiotemporal alignment module and the cross-modal semantic fusion module, forming a complete closed-loop feedback loop. This closed-loop collaborative architecture enables the system to continuously optimize its parameters based on the early warning effect, achieving adaptive adjustment capabilities.

[0026] like Figure 2 As shown, the multi-source data spatiotemporal alignment module is used to receive satellite remote sensing image data, IoT sensor data and social media data from external systems, and to perform spatiotemporal alignment processing on these three types of heterogeneous data through an adaptive spatiotemporal resampling unit, a coordinate unified mapping unit and a timestamp synchronization correction unit, and output spatiotemporally aligned multi-source data for use by subsequent modules.

[0027] The multi-source data spatiotemporal alignment module receives satellite remote sensing imagery, including multispectral remote sensing images and synthetic aperture radar images. This data originates from various satellite platforms, including meteorological satellites, resource satellites, and commercial remote sensing satellites. The raw spatial resolution of the satellite remote sensing imagery ranges from 10 meters to 30 meters, and the temporal resolution ranges from hours to days. The multi-source data spatiotemporal alignment module also receives IoT sensor data, including temperature, humidity, air pressure, wind speed, and precipitation data collected by meteorological sensors; water level, flow rate, and water quality data collected by hydrological sensors; and surface displacement, soil moisture content, and groundwater level data collected by geological sensors. The raw sampling frequency of the IoT sensor data ranges from seconds to minutes. Finally, the multi-source data spatiotemporal alignment module receives social media data, including user-posted disaster descriptions, on-site images, geolocation tags, and posting timestamps. This data originates from public interfaces of social media platforms such as Weibo, WeChat, and Douyin. The posting time of social media data is random and delayed, and geolocation information may contain errors or be missing.

[0028] The adaptive spatiotemporal resampling unit automatically adjusts the resampling parameters based on the original sampling frequency and spatial resolution of each data source, ensuring that data from different data sources are aligned on a unified spatiotemporal grid. The resampling parameters of the adaptive spatiotemporal resampling unit include the temporal resampling interval. and spatial resampling resolution Preferably, the time resampling interval The value ranges from 1 minute to 60 minutes, and the spatial resampling resolution is... The value ranges from 10 meters to 1000 meters. The specific value of the resampling parameter is automatically determined based on the data source type and the severity of the disaster.

[0029] The adaptive spatiotemporal resampling unit employs an adaptive spatiotemporal resampling algorithm to resample data from various sources. For satellite remote sensing image data, bicubic interpolation is used for spatial resampling, and time-weighted averaging is used for temporal resampling. The core calculation process of the adaptive spatiotemporal resampling algorithm is as follows:

[0030] Let the original remote sensing image be located in space. ,time The pixel value at that location is After resampling, in the target space grid Target time The pixel value at that location is Spatial resampling employs a bicubic interpolation kernel function. calculate:

[0031] ,

[0032] in: For spatial resampling at the target location Original time Pixel value at; For the original image at the neighboring pixel location Original time Pixel value at; and The target locations are respectively The x and y coordinates of the surrounding neighboring pixels are taken from the nearest 4×4 neighboring pixels to the target location. It is a bicubic interpolation kernel function. Distance parameter; subscript and These represent the horizontal and vertical neighbor pixel indices, respectively, ranging from -1 to 2. (Bicubic interpolation kernel function) The specific definition is:

[0033] ,

[0034] in: This is the normalized distance between the target location and the locations of neighboring pixels; This is the interpolation parameter, and its preferred value is -0.5.

[0035] Time resampling is calculated using a time-weighted average method:

[0036] ,

[0037] in: For the final resampling at the target location Target time Pixel value at; For spatial resampling at the target location , No. Original time point Pixel value at; For the first The time weights corresponding to each original time point; The number of original time points participating in the time interpolation. Time weight. Calculated using Gaussian kernel function:

[0038] ,

[0039] in: For the target time point; For the first One original time point; The standard deviation parameter of the time Gaussian kernel is determined by the time resampling interval. Adaptive determination, preferred value .

[0040] For IoT sensor data, the adaptive spatiotemporal resampling unit uses a linear interpolation method for time resampling. Let the sensor at the original sampling time point... The measured value at that location is After resampling at the target time The measured value at that location is The calculation formula is:

[0041] ,

[0042] in: For resampling at the target time Sensor measurements at the location; and Target time Two adjacent original sampling time points and Measured value at; subscript Indicates the index of the original sampling time point.

[0043] For social media data, the adaptive spatiotemporal resampling unit employs a spatiotemporal grid aggregation method for resampling. The monitoring area is divided into spatial resolutions. The grid divides the timeline into intervals. Within a time window, social media data falling within the same spatiotemporal grid cell are aggregated. Let the spatiotemporal grid cell be... Contains A collection of social media data, resulting in a feature vector. The following is obtained by averaging the feature vectors of individual data points:

[0044] ,

[0045] in: spatiotemporal grid unit Aggregated social feature vectors; For the first [cell] in this grid cell The original feature vector of each piece of social media data; This represents the number of social media data entries within that grid cell. , , These are the spatial grid horizontal index, vertical index, and time window index, respectively.

[0046] The coordinate unification mapping unit maps spatial data from different coordinate systems to a unified geographic coordinate system. Satellite remote sensing image data typically uses the UTM projected coordinate system or geographic latitude and longitude coordinate system, IoT sensor data uses the geographic latitude and longitude coordinates of the sensor installation location, and social media data uses the geographic latitude and longitude coordinates of GPS positioning or the approximate location of base station positioning. The coordinate unification mapping unit maps the above-mentioned data from different coordinate systems to the WGS84 geographic coordinate system and uses the equal area projection method to transform it into a planar coordinate system for gridding processing.

[0047] The timestamp synchronization correction unit eliminates time delay differences between different data sources. Timestamps from different data sources exhibit systematic delays. The delay in satellite remote sensing imagery data primarily stems from satellite transit time and data download and processing time; the delay in IoT sensor data mainly arises from data acquisition and transmission time; and the delay in social media data primarily arises from the difference between user posting time and event occurrence time. The timestamp synchronization correction unit employs a timestamp synchronization correction algorithm to correct the timestamps of each data source.

[0048] Set data source The original timestamp is The corrected timestamp is The correction formula is:

[0049] ,

[0050] in: For data source Corrected timestamp; For data source The original timestamp; For data source The systemic time delay was obtained through historical data statistics; For data source In time The dynamic time delay deviation at a given point is estimated through time synchronization verification within a sliding window; subscript Indicates the data source index. Represents satellite remote sensing image data, Represents IoT sensor data, This refers to social media data.

[0051] The multi-source data spatiotemporal alignment module outputs spatiotemporally aligned multi-source data, including spatiotemporally aligned remote sensing data, spatiotemporally aligned sensor data, and spatiotemporally aligned social data. These three types of data share a unified spatiotemporal grid coordinate system, providing a foundation for subsequent cross-modal semantic fusion.

[0052] like Figure 3 As shown, the cross-modal semantic fusion module receives the spatiotemporally aligned multi-source data output by the multi-source data spatiotemporal alignment module. It performs deep semantic feature extraction and cross-modal fusion on the multi-source heterogeneous data through remote sensing feature encoder, sensor feature encoder, social semantic encoder and cross-attention fusion unit, and outputs cross-modal fused features for use by the disaster evolution dynamic prediction module.

[0053] The remote sensing feature encoder is used to extract visual feature vectors from spatiotemporally aligned satellite remote sensing image data. The encoder employs a convolutional neural network structure based on residual connections, specifically using the ResNet-50 network architecture as the backbone. The ResNet-50 network includes convolutional layers, batch normalization layers, ReLU activation layers, and residual connection modules, effectively extracting multi-scale visual features from remote sensing images. The encoder processes the input multispectral remote sensing image into a 512-dimensional visual feature vector. Let the input remote sensing image be... ,in For image height, For image width, The number of spectral channels, preferably Corresponding to the four bands: red, green, blue, and near-infrared. The visual feature vector output by the remote sensing feature encoder is represented as follows: Preferably, feature dimension .

[0054] The sensor feature encoder is used to extract temporal feature vectors from spatiotemporally aligned IoT sensor data. It employs a gated recurrent unit (GRU)-based temporal coding structure, effectively capturing the temporal dependencies and trends of sensor data. The GRU controls the flow of information by updating and resetting gates, avoiding the vanishing gradient problem in long-sequence training. The sensor feature encoder processes the input sensor temporal data into a 256-dimensional temporal feature vector. Let the input sensor temporal data be... ,in For timing length, The sensor measures dimensions including parameters such as temperature, humidity, air pressure, wind speed, precipitation, and water level. The temporal feature vector output by the sensor feature encoder is represented as... Preferably, feature dimension .

[0055] The social semantic encoder extracts semantic feature vectors from spatiotemporally aligned social media text and images. It comprises two sub-modules: a text encoder and an image encoder. The text encoder uses a pre-trained language model-based structure, specifically employing the BERT model to semantically encode the social media text, processing it into a 768-dimensional text semantic feature vector. The image encoder uses a visual transformer-based structure, specifically employing the ViT model to visually encode the social media images, processing them into a 768-dimensional visual semantic feature vector. The social semantic encoder then merges the text and visual semantic feature vectors through a concatenation operation into a 512-dimensional comprehensive semantic feature vector. Let the input social media text be... Input social media images as The semantic feature vector output by the social semantic encoder is represented as Preferably, feature dimension .

[0056] The cross-attention fusion unit is used to calculate the correlation weights between feature vectors of different modalities based on the cross-attention mechanism and perform weighted fusion. The cross-attention fusion unit receives the visual feature vectors output by the remote sensing feature encoder. The temporal feature vector output by the sensor feature encoder and the semantic feature vector output by the social semantic encoder Cross-modal fusion features are calculated using a cross-attention fusion algorithm.

[0057] The core computational process of the cross-attention fusion algorithm is as follows. First, the feature vectors of the three modalities are mapped to a unified feature space through a linear transformation: ,in: For the first The original feature vectors of each modality, ; , , The first The query transformation matrix, key transformation matrix, and value transformation matrix for each modality; , , The first Each modality contains a query vector, key vector, and value vector; superscript , , These represent the three transformation types: query, key, and value.

[0058] Then, the cross-attention weights between different modalities are calculated. Taking the attention of the remote sensing modality to the sensing modality and the social modality as an example:

[0059] ,

[0060] ,

[0061] in: The attention weights of remote sensing modes on sensing modes; The attention weights of remote sensing modalities to social modalities; The query vector for the remote sensing modality; and These are the transposes of the key vectors for the sensing modality and the social modality, respectively. The dimension of the key vector is used for scaling to prevent the dot product from becoming too large; The normalized exponential function ensures that the sum of the attention weights is 1. Similarly, the attention weights of the sensing modality and the social modality on other modalities can be calculated.

[0062] Next, the value vectors are weighted and fused based on attention weights:

[0063] ,

[0064] in: This is the fused remote sensing modal feature vector; This represents the original value vector of the remote sensing mode; and These represent the attention weights of the remote sensing mode on the sensing mode and the social mode, respectively. and These are the value vectors of the sensing modality and the social modality, respectively. Similarly, the fused sensing modality feature vector can be calculated. and social modality feature vectors .

[0065] Finally, the three fused modal feature vectors are concatenated and linearly transformed to obtain the final cross-modal fused feature:

[0066] ,

[0067] in: This is the final cross-modal fusion feature vector; To output a linear transformation matrix; This is a vector concatenation operation; , , These are the feature vectors for the fused remote sensing, sensing, and social modalities, respectively. Preferably, the dimension of the cross-modal fused feature is 1024.

[0068] The relevance weights in the cross-attention fusion unit range from 0 to 1. When the relevance weight is lower than a preset fusion threshold, the contribution of the corresponding modal features is suppressed to avoid low-quality or irrelevant modal information interfering with the fusion result. Preferably, the preset fusion threshold is 0.1.

[0069] The cross-modal semantic fusion module outputs cross-modal fusion features, which integrate spatial coverage information from satellite remote sensing, temporal variation information from IoT sensors, and semantic description information from social media, providing a comprehensive multimodal feature representation for dynamic prediction of disaster evolution.

[0070] like Figure 4 As shown, the disaster evolution dynamic prediction module receives the cross-modal fusion features output by the cross-modal semantic fusion module, constructs the fusion features into a spatiotemporal graph structure, and performs message passing and aggregation updates on the node features in the spatiotemporal graph structure through a spatiotemporal graph attention network to predict the spread range, intensity changes and evolution sequence of the disaster, and outputs the disaster evolution prediction results and prediction confidence scores.

[0071] The disaster evolution dynamic prediction module first constructs a spatiotemporal graph structure from cross-modal fusion features. Nodes in the spatiotemporal graph structure represent spatial grid cells, node features are the cross-modal fusion features of the corresponding grid cell, edges connect spatially or temporally adjacent nodes, and edge weights represent the spatiotemporal correlation strength between nodes. Assume the monitoring area is divided into... Each spatial grid unit is divided into time dimensions. If there are 1 time step, then the spacetime graph contains a total of 1000 time steps. The node. The initial feature vectors of the nodes are ,in This represents the cross-modal fusion feature of the spatiotemporal grid cell corresponding to this node.

[0072] Edges in a spatiotemporal graph structure are divided into two categories: spatial edges and temporal edges. Spatial edges connect spatially adjacent nodes within the same time step, using an eight-neighborhood connection method, meaning each node is connected to its eight adjacent nodes in the up, down, left, right, upper-left, upper-right, lower-left, and lower-right directions. Temporal edges connect nodes with the same spatial location in adjacent time steps. The initial weight of spatial edges is calculated based on the spatial distance between nodes.

[0073] ,

[0074] in: For nodes and nodes The initial weights of the spatial edges between them; For nodes and nodes Euclidean distance between corresponding spatial locations; The standard deviation parameter of the Gaussian kernel for spatial distance is preferably set to the spatial grid resolution. The initial weight of the temporal edge is set to 1, indicating that adjacent time steps have equal temporal correlation.

[0075] Spatiotemporal graph attention networks are used for message passing and aggregation updates of node features in a spatiotemporal graph structure. The spatiotemporal graph attention network employs a spatiotemporal graph attention propagation algorithm for multiple rounds of message passing. Each round of message passing includes three steps: message computation, attention weighting, and feature update. The number of message passing rounds in the spatiotemporal graph attention network ranges from 2 to 6 rounds, with 4 rounds being the preferred value.

[0076] The core computational process of the spatiotemporal graph attention propagation algorithm is as follows. In the... In round-robin message passing, the node is calculated first. From its neighboring nodes Received message:

[0077] ,

[0078] in: For the first Wheel Follower Node Passed to node The message vector; For the first The message transformation matrix of the round; For nodes In the The updated feature vectors after each round; For nodes The set of neighboring nodes, including spatial neighbors and temporal neighbors; superscript Indicates the first Round message passing.

[0079] Then, compute the nodes. Attention weights for messages from each neighboring node:

[0080] ,

[0081] in: For the first wheel node For from node Attention weight of messages; This is the attention parameter vector; For the first The attention transformation matrix of the wheel; and They are nodes and nodes In the The updated feature vectors after each round; It is a rectified linear activation function with leakage; This is a vector concatenation operation; superscript This represents the transpose of a vector.

[0082] Next, neighbor messages are weighted and aggregated based on attention weights:

[0083] ,

[0084] in: For the first wheel node Aggregated message feature vector; Attention weights; For nodes and nodes Edge weights between them, spatial edges use The temporal edges use an initial weight of 1; This is the message vector.

[0085] Finally, update the node features:

[0086] ,

[0087] in: For the first After the round of node updates eigenvectors; For the first The update transformation matrix of the cycle; For nodes In the The updated feature vectors after each round; This is the aggregated message feature vector; It is a rectified linear activation function; This is a vector concatenation operation.

[0088] After multiple rounds of message passing, the spatiotemporal graph attention network outputs the final feature vectors of each node. ,in This represents the total number of message rounds. Based on the characteristics of the final node, the disaster evolution dynamic prediction module outputs the disaster evolution prediction results through the prediction head network. The prediction head network includes two branches: the spread range prediction branch and the intensity prediction branch. The spread range prediction branch uses binary classification output to predict whether each grid cell will be affected by the disaster in future time steps. The intensity prediction branch uses regression output to predict the disaster intensity value of each grid cell.

[0089] The formula for predicting the spread range is:

[0090] ,

[0091] in: For nodes The predicted probability of a grid cell being affected by a disaster ranges from 0 to 1. and These are the weight matrix and bias vector of the propagation range prediction branch, respectively; For nodes The final feature vector; This is the Sigmoid activation function.

[0092] The formula for calculating intensity prediction is:

[0093] ,

[0094] in: For nodes The predicted disaster intensity value for the corresponding grid cell; and These are the weight matrix and bias vector of the intensity prediction branch, respectively; For nodes The final eigenvector.

[0095] The disaster evolution dynamic prediction module also outputs a prediction confidence score, which measures the reliability of the prediction results. The prediction confidence score is calculated based on the probability distribution entropy of the spread range prediction and the uncertainty estimate of the intensity prediction.

[0096] ,

[0097] in: The confidence score is calculated from 0 to 1, with a higher value indicating a more reliable prediction. The total number of nodes; For nodes Binary entropy for predicting the spread range When the predicted probability is close to 0 or 1, the entropy value is small, indicating that the prediction is relatively certain and the confidence level is high.

[0098] like Figure 5 As shown, the minute-level early warning generation module receives the disaster evolution prediction results output by the disaster evolution dynamic prediction module, and quickly generates early warning information based on the incremental calculation strategy through the incremental early warning generator. Within the early warning response time threshold, it outputs minute-level early warning information containing disaster type, affected area, early warning level and response suggestions.

[0099] The incremental early warning generator employs an incremental early warning confidence calculation algorithm to achieve minute-level early warning response. The core idea of ​​the incremental calculation strategy is to extract features and update predictions only for newly added data within the time window, reusing intermediate calculation results from historical time windows to avoid redundant calculations on the entire dataset. The early warning response time threshold of the incremental early warning generator ranges from 1 to 10 minutes, with a preferred value of 5 minutes.

[0100] The core calculation process of the incremental early warning confidence calculation algorithm is as follows. Let the current time be... The incremental calculation time window is The historical time window is The incremental time window is Intermediate calculation results within the historical time window are cached, including spatiotemporally aligned historical data, historical fusion features, and historical graph structures. The incremental warning generator only processes newly added data within the incremental time window.

[0101] ,

[0102] in: These are the fusion features after incremental updates; This refers to newly added raw data within the incremental time window; For cached historical fusion features; This is an incremental processing function, including incremental spatiotemporal alignment, incremental feature encoding, and incremental graph update.

[0103] Incremental graph updates employ a local update strategy, updating only the graph nodes and edges related to the newly added data, while keeping the state of other nodes and edges unchanged.

[0104] ,

[0105] in: The updated spacetime graph structure; For cached history graph structure; For the newly added set of nodes; Add a new set of edges; This is a local graph update function.

[0106] Based on the fused features and graph structure after incremental updates, the incremental warning generator performs local message passing through a spatiotemporal graph attention network, updates features only for nodes affected by new data, and then generates warning information through a prediction head network.

[0107] Minute-level early warning information includes the following: disaster type, indicating the predicted disaster category, including floods, earthquakes, landslides, typhoons, etc.; affected area, indicating the spatial range of the predicted disaster impact, expressed in the form of administrative divisions or grid coordinates; warning level, indicating the severity and urgency of the disaster; and response recommendations, indicating emergency response recommendations for the disaster type and warning level.

[0108] Minute-level early warning information includes four levels: blue, yellow, orange, and red. The warning level is determined based on the predicted intensity and scope of the disaster.

[0109] ,

[0110] in: It is at the warning level; To predict the average disaster intensity within the affected area; To predict the area affected by the disaster; , , This is the intensity threshold parameter; , , This is an area threshold parameter. Preferably, it is an intensity threshold parameter. , , Area threshold parameter square kilometers square kilometers square kilometers.

[0111] like Figure 5 As shown, the closed-loop feedback optimization module receives the prediction confidence score output by the disaster evolution dynamic prediction module and the early warning result evaluation data returned by the external feedback verification system. It calculates the comprehensive credibility of the early warning result through the confidence evaluation unit and generates a parameter adjustment signal based on the comprehensive credibility through the parameter dynamic adjustment unit and feeds it back to the front-end module.

[0112] The confidence assessment unit is used to calculate the overall confidence level of the early warning results. The overall confidence level is obtained based on a weighted fusion of the predicted confidence score and the early warning result assessment data. The early warning result assessment data comes from an external feedback verification system and includes the early warning hit rate and the early warning false alarm rate. The early warning hit rate represents the proportion of actual disaster events that are correctly warned by the system, while the early warning false alarm rate represents the proportion of warnings issued by the system that do not actually occur.

[0113] The formula for calculating overall credibility is:

[0114] ,

[0115] in: To ensure overall reliability, the value range is normalized to 0 to 1; The prediction confidence score output by the disaster evolution dynamic prediction module; The accuracy rate of the warning is measured in values ​​ranging from 0 to 1. The false alarm rate is set to a value ranging from 0 to 1. , , For the weighting coefficients, satisfying Preferably, , , That is, prediction confidence and early warning hit rate have equal weight, and the false alarm rate is included in the overall confidence as a negative contribution.

[0116] The parameter dynamic adjustment unit generates spatiotemporal alignment parameter adjustment signals and semantic fusion parameter adjustment signals based on the overall credibility. The unit employs a closed-loop feedback parameter adaptive adjustment algorithm to achieve dynamic parameter optimization.

[0117] The core calculation process of the closed-loop feedback parameter adaptive adjustment algorithm is as follows. Let the current spatiotemporal alignment parameter be... The current semantic fusion parameters are The parameter dynamic adjustment unit adjusts based on overall reliability. The adjustment strategy is determined by comparing the results with the preset confidence threshold.

[0118] when When the system performance is poor, large step sizes should be used to adjust the parameters.

[0119] ,

[0120] ,

[0121] when When the system performance is average, parameters should be adjusted using a medium step size.

[0122] ,

[0123] ,

[0124] when When the system is in good condition, parameters are fine-tuned using small steps.

[0125] ,

[0126] ,

[0127] in: and These are the spatiotemporal alignment parameters before and after adjustment, respectively. and These are the semantic fusion parameters before and after adjustment, respectively; , , These are the learning rate parameters for large step size, medium step size, and small step size, respectively. and These are the gradients of the loss function with respect to the spatiotemporal alignment parameters and the semantic fusion parameters, respectively. and These are the preset lower and upper confidence thresholds, respectively; To optimize the target loss function, it is calculated based on the warning error and confidence bias.

[0128] The adjustment step size of the parameter dynamic adjustment unit is adaptively determined based on the overall confidence level. When the overall confidence level is lower than the preset confidence level lower limit, the first learning rate is used for adjustment. When the overall confidence level is higher than the preset confidence level upper limit, the second learning rate is used for fine adjustment. The first learning rate is greater than the second learning rate. The value range of the first learning rate is 0.005 to 0.02, and the value range of the second learning rate is 0.00005 to 0.0005.

[0129] Preferably, a lower confidence level is preset. Preset credibility limit First learning rate Second learning rate Third learning rate .

[0130] The parameter dynamic adjustment unit sends the spatiotemporal alignment parameter adjustment signal to the multi-source data spatiotemporal alignment module to update the resampling parameters of the adaptive spatiotemporal resampling unit, including the time resampling interval. and spatial resampling resolution The parameter dynamic adjustment unit sends the semantic fusion parameter adjustment signal to the cross-modal semantic fusion module to update the attention weight calculation parameters of the cross-attention fusion unit, including the adjustment values ​​of the query transformation matrix, key transformation matrix, and value transformation matrix.

[0131] Through the parameter dynamic adjustment mechanism of the closed-loop feedback optimization module, the system of the present invention can continuously optimize its own parameters according to the actual early warning effect, realize adaptive adjustment to changes in the external environment and data characteristic drift, and ensure the stability and accuracy of the system in long-term operation.

[0132] The hardware configuration and deployment environment of the real-time fusion early warning system for disaster multi-source data based on deep learning of this invention are as follows.

[0133] In terms of server hardware configuration, the system of this invention is deployed on a high-performance computing server cluster. The main compute node uses dual Intel Xeon Gold 6248 processors, each with 20 physical cores, a clock speed of 2.5GHz, and supports Hyper-Threading technology, providing a total of 80 logical processing units. The main compute node is equipped with 256GB of DDR4 ECC memory, with a memory bandwidth of up to 2933MT / s, meeting the needs of large-scale matrix operations and data caching. The main compute node is equipped with four NVIDIA V100 Tensor Core GPUs, each with 32GB of HBM2 video memory and 5120 CUDA cores, providing approximately 125 TFLOPS of deep learning computing power. The storage system adopts a distributed storage architecture, equipped with a 100TB NVMe SSD storage array, with a read bandwidth of up to 20GB / s and a write bandwidth of up to 15GB / s, meeting the high-speed access requirements of massive remote sensing images and sensor data.

[0134] In terms of network configuration, the server cluster is interconnected via 100Gbps high-speed Ethernet and employs RDMA technology for low-latency data transmission. The external network utilizes a dual-redundancy design, connecting to the satellite remote sensing image data receiving station, IoT sensor gateway, and social media data acquisition server via 10Gbps fiber optic links. Network latency is controlled at the millisecond level, meeting the requirements for real-time data reception.

[0135] In terms of software environment, the operating system is Ubuntu 20.04 LTS server edition, the deep learning framework is PyTorch 1.12, the graph neural network framework is PyTorch Geometric 2.0, and the distributed computing framework is Horovod for multi-GPU parallel training and inference. The database uses PostgreSQL 14 with PostGIS extensions to store geospatial data, InfluxDB to store time-series sensor data, and Elasticsearch to store social media text data. Apache Kafka is used for real-time data stream processing.

[0136] Regarding data interfaces, satellite remote sensing imagery data is acquired through a remote sensing data service interface, supporting multiple satellite data sources such as Landsat, Sentinel, and Gaofen, and supporting standard formats such as GeoTIFF and HDF5. IoT sensor data is received via the MQTT protocol, supporting data encoding formats such as JSON and Protocol Buffers. Social media data is collected through a public API interface, supporting both RESTful interfaces and real-time WebSocket push methods.

[0137] The overall workflow of the real-time fusion and early warning system for multi-source disaster data based on deep learning in this invention is as follows:

[0138] First, the multi-source data spatiotemporal alignment module receives multispectral remote sensing images from satellite remote sensing systems, environmental monitoring data from IoT sensor networks, and user-posted content from social media platforms in real time. The module uses an adaptive spatiotemporal resampling algorithm to uniformly sample data from each data source, a coordinate unification mapping algorithm to convert data from different coordinate systems to a unified geographic coordinate system, and a timestamp synchronization correction algorithm to eliminate time delay differences between data sources, outputting spatiotemporally aligned multi-source data.

[0139] Then, the cross-modal semantic fusion module receives spatiotemporally aligned multi-source data, extracts feature vectors of each modality through remote sensing feature encoder, sensor feature encoder and social semantic encoder respectively, calculates the correlation weight between different modalities through cross-attention fusion algorithm and performs weighted fusion, and outputs cross-modal fused features.

[0140] Next, the disaster evolution dynamic prediction module receives cross-modal fusion features, constructs the fusion features into a spatiotemporal graph structure, performs multiple rounds of message passing and feature updates through the spatiotemporal graph attention propagation algorithm, and outputs the prediction results of disaster spread range, intensity change and evolution sequence through the prediction head network, while also outputting the prediction confidence score.

[0141] Subsequently, the minute-level early warning generation module receives the disaster evolution prediction results and achieves rapid early warning generation through an incremental early warning confidence calculation algorithm. Within the early warning response time threshold, it outputs minute-level early warning information containing disaster type, affected area, early warning level, and response suggestions.

[0142] Finally, the closed-loop feedback optimization module receives the predicted confidence score and the early warning result evaluation data returned by the external feedback verification system. It calculates the comprehensive credibility through the confidence evaluation algorithm, generates the parameter adjustment signal through the closed-loop feedback parameter adaptive adjustment algorithm, and feeds the adjustment signal back to the multi-source data spatiotemporal alignment module and the cross-modal semantic fusion module to optimize the system parameters.

[0143] The technical effectiveness of this invention's real-time disaster multi-source data fusion early warning system was verified in a test environment. The test environment included: a server configured with an Intel Xeon Gold 6248 processor, 256GB of memory, and an NVIDIA V100 GPU; the test dataset included historical disaster monitoring data from 2020 to 2023, covering various disaster types such as floods, earthquakes, landslides, and typhoons, with a total data volume of approximately 1.2TB.

[0144] Spatiotemporal alignment accuracy test results: Compared with the simple spatiotemporal calibration method used in the Chinese invention (CN118278577A), the adaptive spatiotemporal resampling algorithm of this invention improves the spatiotemporal alignment accuracy by about 40%. Specifically, the time alignment error is reduced from an average of 15 minutes to an average of 9 minutes, and the spatial alignment error is reduced from an average of 50 meters to an average of 30 meters.

[0145] Feature fusion completeness test results: Compared with the simple feature splicing method, the cross-attention fusion mechanism of the present invention improves the feature fusion completeness by about 35%. The completeness index is measured by the multimodal feature correlation coefficient. The average correlation coefficient of the method of the present invention is 0.78, while the average correlation coefficient of the simple splicing method is 0.58.

[0146] Prediction accuracy test results: Compared with single deep learning models (such as LSTM and CNN), the spatiotemporal graph attention network of this invention improves the prediction accuracy by about 25%. Specifically, the F1 score for disaster spread range prediction increases from 0.68 to 0.85, and the mean squared error for disaster intensity prediction decreases from 0.12 to 0.09.

[0147] Early warning response time test results: The incremental calculation strategy of the present invention controls the early warning generation time to within 10 minutes, and the average early warning response time is 4.2 minutes, which meets the timeliness requirements of minute-level early warning.

[0148] Closed-loop optimization effect test results: Through continuous parameter adjustment by the closed-loop feedback optimization module, the system maintained a warning hit rate above 85% and a false alarm rate below 15% during long-term operation, ensuring stable system performance. Specifically, during a long-term test lasting 180 days, the system processed approximately 45 million multi-source data records and generated approximately 12,000 warning messages. The closed-loop feedback mechanism automatically performed approximately 350 parameter adjustments during operation, including approximately 50 large-step adjustments, approximately 120 medium-step adjustments, and approximately 180 small-step fine-tuning adjustments. These parameter adjustments enabled the system to adapt to the influence of external factors such as seasonal changes and data source characteristic drift.

[0149] Data flow integrity test results: The data flow processing of the system of this invention exhibits high integrity and traceability. Throughout the entire data processing chain, from raw data access to early warning information output, the data loss rate is less than 0.01%, and the standard deviation of data processing latency is less than 15 seconds, ensuring the reliability and consistency of early warning information. The system supports data traceability of early warning results at any point in time, facilitating early warning effect evaluation and system performance analysis.

[0150] In summary, this invention effectively solves the technical problems of insufficient depth of multi-source heterogeneous data fusion, limited spatiotemporal alignment accuracy, lack of collaborative prediction modeling, insufficient timeliness of early warning, and lack of closed-loop feedback mechanism in the prior art by deeply coupling and coordinating a multi-source data spatiotemporal alignment module, a cross-modal semantic fusion module, a disaster evolution dynamic prediction module, a minute-level early warning generation module, and a closed-loop feedback optimization module. It provides an efficient, accurate, and real-time technical solution for the field of disaster early warning.

[0151] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time fusion and early warning system for multi-source disaster data based on deep learning, characterized in that, include: The multi-source data spatiotemporal alignment module is used to receive satellite remote sensing image data, IoT sensor data and social media data. Through the adaptive spatiotemporal resampling unit, it automatically adjusts the resampling parameters according to the original sampling frequency and spatial resolution of each data source. Through the coordinate unified mapping unit, it maps spatial data from different coordinate systems to a unified geographic coordinate system. Through the timestamp synchronization correction unit, it eliminates the time delay difference between different data sources and outputs spatiotemporally aligned multi-source data. The cross-modal semantic fusion module is used to receive the spatiotemporally aligned multi-source data, extract visual feature vectors from satellite remote sensing image data through remote sensing feature encoder, extract temporal feature vectors from IoT sensor data through sensor feature encoder, extract semantic feature vectors from social media data through social semantic encoder, and calculate the correlation weights between different modal feature vectors based on the cross-attention mechanism through cross-attention fusion unit and perform weighted fusion to output cross-modal fusion features. The disaster evolution dynamic prediction module is used to receive the cross-modal fusion features, construct the fusion features into a spatiotemporal graph structure, perform message passing and aggregation updates on the node features in the spatiotemporal graph structure through a spatiotemporal graph attention network, predict the spread range, intensity changes and evolution sequence of the disaster, and output the disaster evolution prediction results and prediction confidence scores. The minute-level early warning generation module is used to receive the disaster evolution prediction results, and through the incremental early warning generator, it performs partial updates on the new data based on the incremental calculation strategy instead of recalculating the entire data, and generates minute-level early warning information containing disaster type, affected area, early warning level and response suggestions within the early warning response time threshold.

2. The system according to claim 1, characterized in that, It also includes a closed-loop feedback optimization module, which receives the predicted confidence score and the early warning result evaluation data returned by the external feedback verification system. The module calculates the comprehensive credibility of the early warning result through the confidence evaluation unit, and generates spatiotemporal alignment parameter adjustment signals and semantic fusion parameter adjustment signals based on the comprehensive credibility through the parameter dynamic adjustment unit. The spatiotemporal alignment parameter adjustment signals are sent to the multi-source data spatiotemporal alignment module to update the resampling parameters, and the semantic fusion parameter adjustment signals are sent to the cross-modal semantic fusion module to update the attention weights. The comprehensive credibility of the confidence evaluation unit is obtained based on the weighted fusion of the predicted confidence score and the early warning result evaluation data, which includes the early warning hit rate and the early warning false alarm rate.

3. The system according to claim 1, characterized in that, The resampling parameters of the adaptive spatiotemporal resampling unit include a time resampling interval and a spatial resampling resolution. The time resampling interval ranges from 1 minute to 60 minutes, and the spatial resampling resolution ranges from 10 meters to 1000 meters. The resampling parameters are automatically determined based on the data source type and the urgency of the disaster.

4. The system according to claim 1, characterized in that, The relevance weights in the cross-attention fusion unit are obtained by the dot product attention operation of the query vector, key vector, and value vector. The relevance weights range from 0 to 1. When the relevance weights are lower than a preset fusion threshold, the contribution of the corresponding modality features is suppressed.

5. The system according to claim 2, characterized in that, The number of message passing rounds in the spatiotemporal graph attention network ranges from 2 to 6 rounds, and the prediction confidence score ranges from 0 to 1. When the prediction confidence score is lower than the preset confidence threshold, the closed-loop feedback optimization module is triggered to adjust the parameters.

6. The system according to claim 1, characterized in that, The warning response time threshold of the incremental warning generator ranges from 1 minute to 10 minutes. The incremental calculation strategy includes extracting features and updating predictions only for newly added data within the time window, and reusing intermediate calculation results from historical time windows.

7. The system according to claim 2, characterized in that, The adjustment step size of the parameter dynamic adjustment unit is adaptively determined based on the overall confidence level. When the overall confidence level is lower than the preset confidence level lower limit, the first learning rate is used for adjustment. When the overall confidence level is higher than the preset confidence level upper limit, the second learning rate is used for fine adjustment. The first learning rate is greater than the second learning rate. The value range of the first learning rate is 0.005 to 0.02, and the value range of the second learning rate is 0.00005 to 0.0005.

8. The system according to claim 1, characterized in that, The remote sensing feature encoder uses a convolutional neural network structure based on residual connections to extract multi-scale visual features, the sensor feature encoder uses a temporal coding structure based on gated recurrent units to extract temporally dependent features, and the social semantic encoder uses a text encoder based on a pre-trained language model and an image encoder based on a visual transformer to extract textual semantic features and visual semantic features, respectively.

9. The system according to claim 1, characterized in that, The nodes in the spatiotemporal graph structure represent spatial grid cells, and the node features are the cross-modal fusion features of the corresponding grid cells. Edges connect spatially adjacent or temporally adjacent nodes, and the edge weights represent the spatiotemporal correlation strength between nodes.

10. The system according to claim 1, characterized in that, The minute-level early warning information includes four levels: blue, yellow, orange, and red. The warning level is determined based on the disaster prediction intensity and impact range.

Images