Intelligent disaster data acquisition and processing method based on multi-modal crowd sensing

Abstract

The invention belongs to the technical field of artificial intelligence, multi-modal data fusion and disaster sensing, and particularly relates to a disaster situation data intelligent obtaining and processing method based on multi-modal crowd sensing. Comprising a multi-modal crowd-sourcing data acquisition module, a perception scheduling mechanism, a multi-modal confidence propagation algorithm, a perception data denoising module, a core feature extraction and structured information generation module and an update implementation and closed loop feedback mechanism. Different from an existing event extraction method only limited to images or texts, the method establishes a multi-modal information structured generation model, image and video data recognize disaster elements through a target detection model, text data is combined with a natural language processing technology model to extract time, place and event elements, and the event extraction efficiency is improved. The audio data is subjected to voice recognition and sentiment analysis to recognize emergency event keywords, and finally all modal extraction contents are converted into a unified triple structure to support subsequent knowledge graph construction and event traceability analysis of the system.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, multimodal data fusion and disaster perception technology, and specifically to a method for intelligently acquiring and processing disaster data based on multimodal crowd intelligence perception. Background Art

[0002] In recent years, with global climate change and increasing marine environmental instability, the frequency of marine natural disasters such as typhoons, storm surges, and tsunamis has shown a significant upward trend. These disasters not only cause significant damage to marine ecosystems but also pose a serious threat to the lives and property of residents in coastal cities. Especially given the sudden and rapid spread of marine disasters, traditional disaster awareness methods that rely on manual inspections and fixed monitoring points are gradually exposing their limitations, such as delayed response times, severe data silos, limited spatial coverage, and high operating costs.

[0003] To address these shortcomings, crowdsensing technology has gradually attracted the attention of researchers and industry. This technology leverages the participation of large-scale user terminals (such as smartphones, drones, and social networking platforms) to rapidly collect and upload disaster information in complex environments. While crowdsensing offers inherent advantages in terms of large data volumes, wide coverage, and real-time performance, it still faces numerous challenges in practical application.

[0004] First, crowdsensing data is significantly heterogeneous. Data uploaded by different terminals may contain images, videos, audio, or natural language text, with significant differences between modalities. Direct fusion will lead to severe information distortion. Second, crowdsensing data suffers from uncontrollable information quality. User-uploaded data may be highly subjective, have a high false alarm rate, or be completely irrelevant, seriously affecting the accuracy of subsequent disaster identification and response. Furthermore, due to the dynamic and heterogeneous nature of the nodes involved in crowdsensing, their trustworthiness is difficult to quantify and track. Traditional information fusion methods are unable to adapt to the uncertainty and timeliness of this type of data.

[0005] Some current studies have attempted to use deep learning methods (such as convolutional neural networks and recurrent neural networks) to process disaster images or text information, but most work focuses on the recognition and classification of a single modality and lacks the ability to jointly model multimodal data. There is also a lack of credibility modeling and dynamic update mechanisms for uploading nodes. This makes it very easy for the system to make incorrect inferences or false alarms when faced with a large amount of low-quality perception information.

[0006] Therefore, there is an urgent need to build a multimodal crowd-sensing method with dynamic, adaptive and intelligent characteristics, which can effectively integrate multi-source heterogeneous data, automatically identify high-confidence information and output structured disaster situation representation to provide decision support for subsequent disaster response.

[0007] It should be noted that the above content falls within the technical knowledge of the inventor and does not necessarily constitute prior art. Summary of the Invention

[0008] In response to the shortcomings of the existing technology, the present invention provides an intelligent disaster information acquisition method that combines crowd perception and multimodal data processing, proposes a multimodal confidence propagation algorithm, and constructs a complete disaster data intelligent extraction, denoising and structured representation process to solve the problems of poor accuracy of disaster data information, weak processing capabilities, and lack of dynamic trust evaluation in the existing technology.

[0009] To achieve the above objectives, the present invention provides the following technical solution: a method for intelligently acquiring and processing disaster data based on multimodal crowd intelligence perception, comprising:

[0010] The multimodal crowd-intelligence data acquisition module collects disaster information by integrating different types of sensory data, including images, videos, audio, and text. This multimodal data provides a rich source of information for subsequent analysis.

[0011] A perception scheduling mechanism that intelligently schedules perception tasks based on factors such as each perception node's historical accuracy, response time, and quality score, ensuring efficient perception tasks and timely data collection.

[0012] Multimodal confidence propagation algorithm, which integrates data confidence and multimodal data based on graph neural network;

[0013] The sensor data denoising module is used to remove sensor data with large errors and poor quality. Multimodal data is highly heterogeneous and needs to be processed using a denoising algorithm to remove noise and abnormal data.

[0014] The core feature extraction and structured information generation module extracts key feature information from the perception data and converts it into structured data format to facilitate subsequent disaster analysis and decision support;

[0015] Implement an update and closed-loop feedback mechanism. As new data is continuously uploaded and processed, the perception system will continuously update the confidence of the perception nodes and adjust the perception scheduling strategy based on the new data. At the same time, it will continuously clean and optimize the data to ensure that the final output data has high accuracy.

[0016] In some embodiments, the specific process of the multimodal crowd intelligence data acquisition module is as follows:

[0017] Users upload multimodal data in real time through smart devices, including photos of the disaster area, post-disaster videos, descriptions of the disaster on social media, and audio data from the disaster site. Each set of data is accompanied by a timestamp, location identifier, and device feature label, forming a set of multimodal perception data;

[0018] The system standardizes the data through location positioning, timestamp and device feature tags to form the original input of multimodal perception data.

[0019] In some embodiments, the specific process of the perception scheduling mechanism is as follows:

[0020] The system assigns a scheduling score to each sensing node by analyzing the accuracy, response time, and data quality of historical data uploads;

[0021] Nodes with higher scores are selected first to collect disaster information in key areas;

[0022] The specific scheduling process is sorted by calculating the perception score of each node:

[0023] S i =α·accuracy(i)+β·response_time(i)+γ·quality(i)

[0024] Among them, S i is the scheduling score of node i, accuracy(i) represents the accuracy of the data of node i, response_time(i) is the response time, quality(i) represents the data quality score, and α, β, γ are weight coefficients.

[0025] In some embodiments, the specific process of the multimodal belief propagation algorithm is as follows:

[0026] By evaluating the confidence of the initial data of each perception node, the confidence is propagated among the perception nodes through the graph structure;

[0027] In this graph structure, nodes represent sensing terminals, and edges represent the similarities between nodes. The graph neural network iteratively updates the confidence value of each node, ultimately assigning each node a confidence score that indicates the reliability of its uploaded data.

[0028] All sensor nodes are regarded as nodes in the graph, and each node is initially assigned a confidence level p according to the data it uploads. i , the confidence value indicates the preliminary reliability of the node data, and the edge weight between nodes indicates the similarity between different nodes. The edge weight calculation formula is as follows:

[0029]

[0030] Among them, x i and x y is the feature vector of nodes i and j, σ is a hyperparameter used to control the width of the similarity measure;

[0031] In addition, graph neural networks are used for confidence propagation. After each propagation, the confidence of a node is updated to the weighted average of the confidences of its adjacent nodes:

[0032]

[0033] in, is the confidence of node i after the first round of propagation, N(i) represents the set of adjacent nodes of node i, and W ij It is the edge right. is the confidence of neighboring node j in the tth round of propagation.

[0034] In some embodiments, the specific process of the perception data denoising module is as follows:

[0035] Combining modal consistency detection with the isolation forest algorithm to identify and exclude low-confidence and abnormal data. The isolation forest algorithm is used to detect and remove outliers that significantly deviate from the data distribution.

[0036] Modal consistency testing is used to evaluate the consistency between different modes. If the consistency is low, the data is considered to be of low quality and will be rejected.

[0037] The Isolation Forest algorithm is used to identify and remove outliers. Multiple decision trees are constructed and outliers are determined based on the depth of the tree. If a data point is separated at a shallow depth in the tree, it is an outlier and needs to be processed.

[0038] In the process of multimodal data processing, conflicts may exist between different modalities. Therefore, modal consistency assessment is used to detect the reliability of data. The correlation between different modal data is calculated and the consistency is evaluated using the correlation score. The specific formula is as follows:

[0039]

[0040] Among them, m i and m j are the eigenvectors of mode i and mode j,

[0041] Cov is the covariance, Var is the variance, if the consistency score C ij If the value is lower than the set threshold, the data is considered to be noise and needs to be eliminated.

[0042] In some embodiments, the core feature extraction and structured information generation module has the following specific process:

[0043] Image and video feature extraction: Target detection algorithms are used on image data to identify disaster-related targets and extract relevant location information. For video data, each frame is processed through time series analysis to extract key frame information and changes between adjacent frames.

[0044] Text feature extraction: Using natural language processing technology on text data to perform named entity recognition and relationship extraction, extracting key information, including the name of the disaster area, event type, and the event that occurred;

[0045] Audio feature extraction: Using language recognition technology on audio data, converting the audio into text, and then extracting disaster-related information through sentiment analysis;

[0046] The extracted image, text, and audio information are fused into a unified structured data format, usually a triple. The structured data is represented as:

[0047] Event i =(event_type i ,location i ,time i )

[0048] Among them, event_type i Indicates the disaster type, location i Is the place where it happened, time i is the time when the event occurred.

[0049] Compared with the existing technology, the present invention provides a method for intelligent acquisition and processing of disaster data based on multimodal crowd intelligence perception, which has the following beneficial effects:

[0050] This method for intelligently acquiring and processing disaster data based on multimodal crowdsensing abstracts crowdsensing nodes into a graph structure, where nodes represent the terminal devices that upload data and edges represent the semantic or spatial similarities between different devices. This graph structure is modeled and learned through a graph neural network to achieve multi-level propagation and fusion of confidence levels, overcoming the limitations of traditional methods that rely on independent judgment of data credibility. By introducing an iterative propagation mechanism, the method not only improves the dynamic assessment capability of data quality but also achieves fault-tolerant suppression of overall decision-making caused by local anomalies.

[0051] Traditional crowd-based intelligence scheduling often focuses on balanced allocation, making it difficult to take into account the historical performance of nodes. This invention introduces a perception scoring function that integrates the accuracy, response speed, and content quality of node uploaded data to perform scoring and ranking, thereby constructing a dynamically adjustable scheduling strategy. This mechanism can prioritize nodes with good historical performance and stable data quality to perform key area perception tasks, fundamentally optimizing the efficiency of system resource allocation.

[0052] To address the difficulty of semantic alignment between multimodal data, the present invention constructs a multimodal fusion mechanism based on confidence propagation. First, the confidence is calculated through initial quality assessment of different modalities. Then, with the help of graph neural networks, the confidence is transferred between modalities and nodes and iteratively updated, ultimately forming a multimodal joint confidence score to achieve reliability enhancement under heterogeneous information fusion. In addition, this method introduces a modal consistency detection mechanism to identify and eliminate conflicting data by calculating the covariance and correlation between modal features.

[0053] To improve the robustness of the perception system, this paper introduces the isolation forest algorithm as an anomaly detection tool. By constructing multiple random tree models, the distribution characteristics of the input data are deeply mined and learned, achieving automatic detection and cleaning of low-quality and deviated data. Combined with modal consistency assessment, a complete multimodal data quality control process is formed to ensure high-quality input for the subsequent structured feature extraction process.

[0054] Unlike existing event extraction methods that rely solely on images or text, this method establishes a structured generative model for multimodal information. Image and video data are analyzed using an object detection model to identify disaster elements. Text data is combined with the BERT model to extract time, location, and event elements. Audio data is then analyzed using speech recognition and sentiment analysis to identify emergency keywords. Ultimately, the extracted content from all modalities is converted into a unified triple structure (event type, time, and location), supporting the system's subsequent knowledge graph construction and event tracing analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flowchart of the intelligent disaster information acquisition and data processing technology implemented in the present invention. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention and the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] It should be understood that the step numbers used herein are only for convenience of description and are not intended to limit the order in which the steps are executed.

[0058] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0059] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0060] The term "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items.

[0061] See also Figure 1 In this embodiment:

[0062] A method for intelligently acquiring and processing disaster data based on multimodal crowd sensing, comprising:

[0063] The multimodal crowd-intelligence data acquisition module collects disaster information by integrating different types of sensory data, including images, videos, audio, and text. This multimodal data provides a rich source of information for subsequent analysis.

[0064] The specific process of the multimodal crowd intelligence data collection module is as follows:

[0065] Users upload multimodal data in real time through smart devices, including photos of the disaster area, post-disaster videos, descriptions of the disaster on social media, and audio data from the disaster site. Each set of data is accompanied by a timestamp, location identifier, and device feature label, forming a set of multimodal perception data;

[0066] The system normalizes the data through location positioning, timestamps, and device feature tags to form the original input of multimodal perception data;

[0067] A perception scheduling mechanism that intelligently schedules perception tasks based on factors such as each perception node's historical accuracy, response time, and quality score, ensuring efficient perception tasks and timely data collection.

[0068] The specific process of the perception scheduling mechanism is as follows:

[0069] The system assigns a scheduling score to each sensing node by analyzing the accuracy, response time, and data quality of historical data uploads;

[0070] Nodes with higher scores are selected first to collect disaster information in key areas;

[0071] The specific scheduling process is sorted by calculating the perception score of each node:

[0072] S i =α·accuracy(i)+β·response_time(i)+γ·quality(i)

[0073] Among them, S i is the scheduling score of node i, accuracy(i) represents the accuracy of the data of node i, response_time(i) is the response time, quality(i) represents the data quality score, and α, β, γ are weight coefficients;

[0074] Multimodal confidence propagation algorithm, which integrates data confidence and multimodal data based on graph neural network;

[0075] The specific process of the multimodal belief propagation algorithm is as follows:

[0076] By evaluating the confidence of the initial data of each perception node, the confidence is propagated among the perception nodes through the graph structure;

[0077] In this graph structure, nodes represent sensing terminals, and edges represent the similarities between nodes. The graph neural network iteratively updates the confidence value of each node, ultimately assigning each node a confidence score that indicates the reliability of its uploaded data.

[0078] All sensor nodes are regarded as nodes in the graph, and each node is initially assigned a confidence level p according to the data it uploads. i , the confidence value indicates the preliminary reliability of the node data, and the edge weight between nodes indicates the similarity between different nodes. The edge weight calculation formula is as follows:

[0079]

[0080] Among them, x i and x y is the feature vector of nodes i and j, σ is a hyperparameter used to control the width of the similarity measure;

[0081] In addition, graph neural networks are used for confidence propagation. After each propagation, the confidence of a node is updated to the weighted average of the confidences of its adjacent nodes:

[0082]

[0083] in, is the confidence of node i after the first round of propagation, N(i) represents the set of adjacent nodes of node i, and W ij It is the edge right. is the confidence of neighboring node j in the tth round of propagation;

[0084] The sensor data denoising module is used to remove sensor data with large errors and poor quality. Multimodal data is highly heterogeneous and needs to be processed using a denoising algorithm to remove noise and abnormal data.

[0085] The specific process of the perception data denoising module is as follows:

[0086] Combining modal consistency detection with the isolation forest algorithm to identify and exclude low-confidence and abnormal data. The isolation forest algorithm is used to detect and remove outliers that significantly deviate from the data distribution.

[0087] Modal consistency testing is used to evaluate the consistency between different modes. If the consistency is low, the data is considered to be of low quality and will be rejected.

[0088] The Isolation Forest algorithm is used to identify and remove outliers. Multiple decision trees are constructed and outliers are determined based on the depth of the tree. If a data point is separated at a shallow depth in the tree, it is an outlier and needs to be processed.

[0089] In the process of multimodal data processing, conflicts may exist between different modalities. Therefore, modal consistency assessment is used to detect the reliability of data. The correlation between different modal data is calculated and the consistency is evaluated using the correlation score. The specific formula is as follows:

[0090]

[0091] Among them, m i and m j are the eigenvectors of mode i and mode j,

[0092] Cov is the covariance, Var is the variance, if the consistency score C ij If the value is lower than the set threshold, the data is considered to be noise and needs to be eliminated;

[0093] The core feature extraction and structured information generation module extracts key feature information from the perception data and converts it into structured data format to facilitate subsequent disaster analysis and decision support;

[0094] The specific process of the core feature extraction and structured information generation module is as follows:

[0095] Image and video feature extraction: The Faster R-CNN target detection algorithm is used on image data to identify disaster-related targets and extract relevant location information. For video data, each frame is processed through time series analysis to extract key frame information and changes between adjacent frames.

[0096] Text feature extraction: Using NLP natural language processing technology on text data, the BERT model is selected for named entity recognition and relationship extraction to extract key information, including the name of the disaster area, event type, and the event that occurred;

[0097] Audio feature extraction: Using language recognition technology on audio data, converting the audio into text, and then using sentiment analysis to extract disaster-related information, such as words like "urgent" and "serious";

[0098] The extracted image, text, and audio information are integrated into a unified structured data format, usually a triple (event type, location, time). The structured data is represented as:

[0099] Event i =(event_type i ,location i ,time i )

[0100] Among them, event_type i Indicates the disaster type, location i Is the place where it happened, time i is the time when the event occurred;

[0101] Implement an update and closed-loop feedback mechanism. As new data is continuously uploaded and processed, the perception system will continuously update the confidence of the perception nodes and adjust the perception scheduling strategy based on the new data. At the same time, it will continuously clean and optimize the data to ensure that the final output data has high accuracy.

[0102] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are described briefly because they are generally similar to the method embodiments. For relevant parts, refer to the description of the method embodiments.

[0103] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for intelligent acquisition and processing of disaster data based on multimodal crowd intelligence perception, characterized in that: include: The multimodal crowd-intelligence data collection module collects disaster information by integrating different types of sensory data, including images, videos, audio, and text. This multimodal data provides a rich source of information for subsequent analysis. A sensing scheduling mechanism that intelligently schedules sensing tasks based on factors such as each sensing node's historical accuracy, response time, and quality score, ensuring efficient sensing tasks and timely data collection. Multimodal confidence propagation algorithm, which integrates data confidence and multimodal data based on graph neural network; The sensor data denoising module is used to remove sensor data with large errors and poor quality. Multimodal data is highly heterogeneous and needs to be processed using a denoising algorithm to remove noise and abnormal data. The core feature extraction and structured information generation module extracts key feature information from the perception data and converts it into structured data format to facilitate subsequent disaster analysis and decision support; Implement an update and closed-loop feedback mechanism. As new data is continuously uploaded and processed, the perception system will continuously update the confidence of the perception nodes and adjust the perception scheduling strategy based on the new data. At the same time, it will continuously clean and optimize the data to ensure that the final output data has high accuracy.

2. The method for intelligently acquiring and processing disaster data based on multimodal crowd intelligence perception according to claim 1 is characterized in that: The specific process of the multimodal crowd intelligence data acquisition module is as follows: Users upload multimodal data in real time through smart devices, including photos of the disaster area, post-disaster videos, descriptions of the disaster on social media, and audio data from the disaster site. Each set of data is accompanied by a timestamp, location identifier, and device feature label, forming a set of multimodal perception data; The system standardizes the data through location positioning, timestamp and device feature tags to form the original input of multimodal perception data.

3. The method for intelligently acquiring and processing disaster data based on multimodal crowd intelligence perception according to claim 1 is characterized in that: The specific process of the perception scheduling mechanism is as follows: The system assigns a scheduling score to each sensing node by analyzing the accuracy, response time, and data quality of historical data uploads; Nodes with higher scores are selected first to collect disaster information in key areas; The specific scheduling process is sorted by calculating the perception score of each node: S i =α·accuracy(i)+β·response_time(i)+γ·quality(i) Among them, S i is the scheduling score of node i, accuracy(i) represents the accuracy of the data of node i, response_time(i) is the response time, quality(i) represents the data quality score, and α, β, γ are weight coefficients.

4. The method for intelligently acquiring and processing disaster data based on multimodal crowd intelligence perception according to claim 1 is characterized in that: The specific process of the multimodal belief propagation algorithm is as follows: By evaluating the confidence of the initial data of each perception node, the confidence is propagated among the perception nodes through the graph structure; In this graph structure, nodes represent sensing terminals, and edges represent the similarities between nodes. The graph neural network iteratively updates the confidence value of each node, ultimately assigning each node a confidence score that indicates the reliability of its uploaded data. All sensor nodes are regarded as nodes in the graph, and each node is initially assigned a confidence level p according to the data it uploads. i , the confidence value indicates the preliminary reliability of the node data, and the edge weight between nodes indicates the similarity between different nodes. The edge weight calculation formula is as follows: Among them, x i and x y is the feature vector of nodes i and j, σ is a hyperparameter used to control the width of the similarity measure; In addition, graph neural networks are used for confidence propagation. After each propagation, the confidence of a node is updated to the weighted average of the confidences of its adjacent nodes: in, is the confidence of node i after the first round of propagation, N(i) represents the set of adjacent nodes of node i, and W ij It is the edge right. is the confidence of neighboring node j in the tth round of propagation.

5. The method for intelligently acquiring and processing disaster data based on multimodal crowd intelligence perception according to claim 1 is characterized in that: The specific process of the perception data denoising module is as follows: Combining modal consistency detection with the isolation forest algorithm to identify and exclude low-confidence and abnormal data. The isolation forest algorithm is used to detect and remove outliers that significantly deviate from the data distribution. Modal consistency testing is used to evaluate the consistency between different modes. If the consistency is low, the data is considered to be of low quality and will be rejected. The Isolation Forest algorithm is used to identify and remove outliers. Multiple decision trees are constructed and outliers are determined based on the depth of the tree. If a data point is separated at a shallow depth in the tree, it is an outlier and needs to be processed. In the process of multimodal data processing, conflicts may exist between different modalities. Therefore, modal consistency assessment is used to detect the reliability of data. The correlation between different modal data is calculated and the consistency is evaluated using the correlation score. The specific formula is as follows: Among them, m i and m j are the eigenvectors of mode i and mode j, Cov is the covariance, Var is the variance, if the consistency score C ij If the value is lower than the set threshold, the data is considered to be noise and needs to be eliminated.

6. The method for intelligently acquiring and processing disaster data based on multimodal crowd intelligence perception according to claim 1 is characterized in that: The specific process of the core feature extraction and structured information generation module is as follows: Image and video feature extraction: Target detection algorithms are used on image data to identify disaster-related targets and extract relevant location information. For video data, each frame is processed through time series analysis to extract key frame information and changes between adjacent frames. Text feature extraction: Using natural language processing technology on text data to perform named entity recognition and relationship extraction, extracting key information, including the name of the disaster area, event type, and the event that occurred; Audio feature extraction: Using language recognition technology on audio data, converting the audio into text, and then extracting disaster-related information through sentiment analysis; The extracted image, text, and audio information are integrated into a unified structured data format, usually a triple (event type, location, time). The structured data is represented as: Event i =(event_type i ,location i ,time i ) Among them, event_type i Indicates the disaster type, location i Is the place where it happened, time i is the time when the event occurred.

Images