Natural disaster management system based on multi-source information fusion
Through the natural disaster management system with multi-source information fusion and Kafka+Storm architecture, the problem of low data processing efficiency in the existing technology is solved, high throughput and real-time performance is achieved, and the accuracy and response efficiency of disaster warning are improved.
Patent Information
- Application Number
- CN202410906254.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-08
- Publication Date
- 2024-11-29
AI Technical Summary
When the existing natural disaster monitoring and early warning system processes massive information from multiple sensors and data sources, the data processing efficiency is low, resulting in a long warning processing time, making it difficult to quickly extract useful information and conduct effective analysis.
Adopt a natural disaster management system based on multi-source information fusion, and collects meteorological, geological and hydrological data through the data acquisition module. The data processing module stores the data in the data warehouse and classifies and standardizes it. The data analysis module uses the distributed version of Kafka and Storm. The computing architecture is processed in parallel, combining Kafka, Spark and Hadoop HDFS for data access, analysis and storage, real-time data processing and early warning.
It improves data processing efficiency, achieves high throughput and real-time performance per unit time, can quickly extract useful information and conduct effective analysis, shorten the warning processing time, and improves the accuracy and response efficiency of disaster warnings.
Smart Images

Figure CN119046318A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a natural disaster management system based on multi-source information fusion. Background Art
[0002] At present, most systems are used to complete the monitoring and early warning tasks of natural disasters, which leads to inconvenient management and high maintenance costs;
[0003] At the same time, in the disaster monitoring and early warning system, a large amount of information from different sensors and data sources needs to be processed in real time. The size of the data throughput directly affects whether the system can quickly extract useful information from the massive data and conduct effective analysis and prediction;
[0004] Therefore, there is an urgent need for a solution that can improve data processing efficiency per unit time, thereby effectively shortening the time required for the system to perform early warning processing. Summary of the invention
[0005] In view of the technical defects mentioned in the background technology, an object of the embodiments of the present invention is to provide a natural disaster management system based on multi-source information fusion to improve data processing efficiency.
[0006] To achieve the above object, an embodiment of the present invention provides a natural disaster management system based on multi-source information fusion, the system comprising:
[0007] A data acquisition module, used for collecting multi-source data; wherein the multi-source data includes meteorological data, geological data and hydrological data;
[0008] A data processing module, used to store the multi-source data in a data warehouse and perform classification, standardization and archiving processing;
[0009] The data analysis module is used to process the processed data in parallel using a preset distributed computing framework to improve data processing efficiency; wherein the distributed computing framework adopts the architecture of Kafka and Storm, the data in Kafka is connected to the Storm cluster, and the nodes in the Storm cluster are also deployed with trained disaster warning models.
[0010] As a specific implementation of the present application, the parallel processing of data specifically includes:
[0011] Kafka Aspout consumes data and sets the number of concurrent data processing of Bolt according to the processing volume of each category of data;
[0012] Set the warning threshold corresponding to the impact factor, use Kafka to consume data in real time, Storm to process data concurrently, monitor data changes in real time, match and process according to the warning threshold, and issue a real-time warning when abnormal data is found;
[0013] Stream processing accumulates data calculations, combines Storm time window calculation rules, sets thresholds for different time periods, and performs real-time accumulation calculation warnings.
[0014] As a specific implementation of the present application, the impact factor performs data access, analysis, processing and storage based on Kafka, Spark and Hadoop HDFS.
[0015] As a specific implementation of the present application, data transmission between modules is performed through Kafka message middleware to reduce the coupling between modules;
[0016] At the same time, different data types are expanded through Kafka topic partitions, thus ensuring that the system has the advantages of stable openness and scalability.
[0017] As a preferred implementation of the present application, during the early warning, the data analysis module also evaluates the corresponding disaster impact range based on the analysis of external data; wherein the external data includes terrain and building structure data;
[0018] During the emergency response phase, the disaster warning model is also used to guide the deployment of rescue teams and the distribution of materials to improve rescue efficiency.
[0019] As a specific implementation of the present application, there are multiple disaster warning models, including a rainstorm monitoring and flood prevention and control model, a snowstorm monitoring model and a road icing monitoring model.
[0020] As a specific implementation method of the present application, when the multi-source data is collected, a preset collection method is used and combined with real-time collection over a 5G network; wherein the collection method includes satellite collection, ground measurement, and aerial photography.
[0021] As a specific implementation of the present application, Hadoop's big data distributed storage technology is used for data storage.
[0022] As a specific implementation method of the present application, the analyzed warning data is written into Kafka and pushed to a third-party platform or displayed through a visual large screen.
[0023] The technical solution provided by the embodiment of the present invention can achieve high throughput by collecting and processing multi-source data and adopting the distributed computing frame Kafka+Storm architecture to receive and process data in real time, so as to be convenient for continuous computing and continuous query of data streams, thereby realizing the real-time and reliability of data analysis; thereby meeting the needs of improving the ability to process data and the efficiency of data processing within a unit time. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.
[0025] Figure 1 is a structural diagram of a natural disaster management system based on multi-source information fusion provided by an embodiment of the present invention;
[0026] Figure 2 It is a schematic diagram of the architecture of Kafka and Storm provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0029] Producer: producer;
[0030] Consumer: consumer, user;
[0031] Topology: topology structure;
[0032] Kafka: achieves high throughput, persistence, and distributed data storage, and has the advantages of easy scalability and convenient client status maintenance;
[0033] Storm: Based on memory computing, it receives network data sources for real-time computing;
[0034] Zookeeper: zk is highly synchronized and scalable, and the messages in zk are ordered, reliable, atomic, and real-time;
[0035] Hadoop: strong scalability, high reliability, and high fault tolerance.
[0036] Please refer to Figure 1 to Figure 2 , an embodiment of the present invention provides a natural disaster management system based on multi-source information fusion, the system comprising:
[0037] A data acquisition module, used for collecting multi-source data; wherein the multi-source data includes meteorological data, geological data and hydrological data;
[0038] A data processing module, used to store the multi-source data in a data warehouse and perform classification, standardization and archiving processing;
[0039] The data analysis module is used to process the processed data in parallel using a preset distributed computing framework to improve data processing efficiency; wherein the distributed computing framework adopts the architecture of Kafka and Storm, the data in Kafka is connected to the Storm cluster, and the nodes in the Storm cluster are also deployed with trained disaster warning models.
[0040] In this embodiment, when the multi-source data is collected, a preset collection method is used in combination with real-time collection over a 5G network; wherein the collection method includes satellite collection, ground measurement, and aerial photography, thereby realizing the integration of meteorological, geological, hydrological, and other multi-source data;
[0041] At the same time, historical data related to early warning is also collected;
[0042] And clean and standardize the collected data to ensure data quality and consistency; including deduplication, desensitization and other processing.
[0043] Furthermore, when storing data, Hadoop’s big data distributed storage technology is used to achieve storage and analysis of structured / semi-structured big data through Hadoop expansion and encapsulation;
[0044] In this embodiment, the parallel processing of data specifically includes:
[0045] Kafka Aspout consumes data and sets the number of concurrent data processing of Bolt according to the processing volume of each category of data;
[0046] In a Storm topology, you can control whether KafkaSpout runs in multithreaded mode by setting its parallelism; parallelism refers to the number of KafkaSpout instances running simultaneously in the topology;
[0047] KafkaSpout in multi-threaded mode can provide higher throughput and concurrent processing capabilities. Each KafkaSpout instance reads data from Kafka independently and distributes it to downstream components for processing. This can achieve parallel processing of data and improve the processing speed of the entire topology.
[0048] Set the warning threshold corresponding to the impact factor, use Kafka to consume data in real time, Storm to process data concurrently, monitor data changes in real time, match and process according to the warning threshold, and issue a real-time warning when abnormal data is found;
[0049] Stream processing accumulates data calculations, combines Storm time window calculation rules, sets thresholds for different time periods, and performs real-time accumulation calculation warnings;
[0050] The time window is a preset time range. By accumulating the values within the time range and then matching them with the associated threshold, it is determined whether to issue an early warning. For example, an early warning is issued after exceeding the threshold.
[0051] The time window in Storm is an important mechanism for processing time-related events or data streams. Storm provides several different time windows, including Tumbling Window, Sliding Window, and Session Window.
[0052] Storm makes it easy to write and scale complex real-time computations on a computer cluster, ensuring that every message is processed, and quickly processing millions of messages per second on a small cluster.
[0053] Specifically, the impact factor performs data access, analysis, processing and storage based on Kafka, Spark and Hadoop HDFS;
[0054] The influencing factors are the influencing factors corresponding to the generation of various natural disasters, such as rainfall, river water level, wind force, dilapidated buildings, etc.
[0055] During the processing, the data parameters associated with the influencing factors are combined for analysis; for example, the rainfall is combined with the wind speed and the dangerous conditions of the surrounding houses for processing.
[0056] It should be noted that there are multiple disaster warning models, including a rainstorm monitoring and flood prevention and control model, a snowstorm monitoring model and a road icing monitoring model.
[0057] During implementation, these models comprehensively consider information from multiple data sources such as remote sensing, meteorology, and geology, and can more accurately simulate and predict the occurrence and development of natural disasters; the corresponding models can be constructed through machine learning, deep learning, statistical analysis and other methods.
[0058] Furthermore, the modules transmit data through Kafka message middleware to reduce the coupling between modules;
[0059] At the same time, different data types are expanded through Kafka topic partitions, thus ensuring that the system has the advantages of stable openness and scalability.
[0060] It should be noted that the extensions include topic extensions and partition extensions; a topic can correspond to multiple partitions, and each partition is associated with a corresponding thread; different topics can be created according to different industries or data types, which not only ensures data security but also improves the corresponding processing speed.
[0061] This solution uses the Kafka+Storm solution to process real-time and uninterrupted data. The advantage of Storm is that the status of nimbus (control node) and worker (working node) are stored in zookeeper. Therefore, when the worker goes down, it can be quickly restored through zookeeper, ensuring the timeliness and reliability of big data analysis.
[0062] From the above description, it can be seen that the technical solution provided by the embodiment of the present invention can achieve high throughput through the collection and processing of multi-source data and the real-time reception and processing of data using the distributed computing frame Kafka+Storm architecture, which is convenient for continuous computing and continuous query of data streams, thereby achieving real-time and reliability of data analysis; thereby meeting the needs of improving the ability to process data and the efficiency of data processing within a unit time.
[0063] Furthermore, by inputting multi-source data collected in real time into the disaster warning model, the system can quickly assess the current disaster situation and issue corresponding warning information. For example, in heavy rain and snow monitoring, by real-time analysis of rainfall and snowfall data, the model can quickly determine key parameters such as road waterlogging, reservoir flood control, traffic conditions, emergency material allocation, and crop protection, providing an important basis for local warning.
[0064] During the early warning, the data analysis module also evaluates the corresponding disaster impact range based on the analysis of external data; wherein the external data includes terrain and building structure data;
[0065] During the emergency response phase, the disaster warning model is also used to guide the deployment of rescue teams and the distribution of materials to improve rescue efficiency.
[0066] In this embodiment, the analyzed warning data will also be written into Kafka and pushed to a third-party platform or displayed through a visual large screen; including publishing to multiple channels such as Web platforms and mobile applications to ensure information coverage and timeliness; and achieve real-time push to regulatory authorities to enable monitoring, warning and quick disposal operations.
[0067] In the embodiments provided in the present application, it should be understood that the disclosed system can also be implemented in other ways. The system embodiments described above are merely schematic. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0068] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.
[0069] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
[0070] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A natural disaster management system based on multi-source information fusion, characterized in that: The system comprises: A data acquisition module, used for collecting multi-source data; wherein the multi-source data includes meteorological data, geological data and hydrological data; A data processing module, used to store the multi-source data in a data warehouse and perform classification, standardization and archiving processing; The data analysis module is used to process the processed data in parallel using a preset distributed computing framework to improve data processing efficiency; wherein the distributed computing framework adopts the architecture of Kafka and Storm, the data in Kafka is connected to the Storm cluster, and the nodes in the Storm cluster are also deployed with trained disaster warning models.
2. A natural disaster management system based on multi-source information fusion as claimed in claim 1, characterized in that: The parallel processing of data specifically includes: Kafka Aspout consumes data and sets the number of concurrent data processing of Bolt according to the processing volume of each category of data; Set the warning threshold corresponding to the impact factor, use Kafka to consume data in real time, Storm to process data concurrently, monitor data changes in real time, match and process according to the warning threshold, and issue a real-time warning when abnormal data is found; Stream processing accumulates data calculations, combines Storm time window calculation rules, sets thresholds for different time periods, and performs real-time accumulation calculation warnings.
3. A natural disaster management system based on multi-source information fusion as claimed in claim 2, characterized in that: The impact factor is based on Kafka, Spark and Hadoop HDFS for data access, analysis, processing and storage.
4. A natural disaster management system based on multi-source information fusion as claimed in any one of claims 1 to 3, characterized in that: The modules transmit data through Kafka message middleware to reduce the coupling between modules. At the same time, different data types are expanded through Kafka topic partitions, thus ensuring that the system has the advantages of stable openness and scalability.
5. A natural disaster management system based on multi-source information fusion as claimed in claim 4, characterized in that: During the early warning, the data analysis module also evaluates the corresponding disaster impact range based on the analysis of external data; wherein the external data includes terrain and building structure data; During the emergency response phase, the disaster warning model is also used to guide the deployment of rescue teams and the distribution of materials to improve rescue efficiency.
6. A natural disaster management system based on multi-source information fusion as claimed in claim 1, characterized in that: There are multiple disaster warning models, including a rainstorm monitoring and flood prevention and control model, a snowstorm monitoring model and a road icing monitoring model.
7. A natural disaster management system based on multi-source information fusion as claimed in claim 1, characterized in that: When the multi-source data is collected, a preset collection method is used in combination with real-time collection over a 5G network; wherein the collection method includes satellite collection, ground measurement, and aerial photography.
8. A natural disaster management system based on multi-source information fusion as claimed in claim 1, characterized in that: When storing data, Hadoop's big data distributed storage technology is used.
9. A natural disaster management system based on multi-source information fusion as claimed in claim 2, characterized in that: The analyzed warning data is written into Kafka and pushed to a third-party platform or displayed through a visual large screen.
Citation Information
Cited By
Industrial internet of things distributed data processing method and middle station
CN120321239A
