Multi-modal knowledge graph construction method and expert system for flood disaster dangerous case identification, judgment and disposal

Abstract

The invention relates to a multi-modal knowledge graph construction method and expert system for flood disaster dangerous case recognition, judgment and disposal, and the method comprises the steps: obtaining multi-modal flood dangerous case data which comprises two or more of text data, image data, video data and audio data; flood control knowledge extraction is carried out according to the multi-mode flood dangerous case data; and according to the flood control knowledge triple, performing multi-modal flood dangerous case knowledge fusion and flood dangerous case knowledge expression to obtain a multi-modal knowledge graph for flood control. According to the method, flood danger knowledge extraction, multi-modal flood danger knowledge fusion and flood control knowledge expression are carried out by acquiring flood danger data of different modals, a comprehensive multi-modal knowledge graph special for the field of water conservancy flood control is constructed, reliable data guidance and support can be provided for flood control personnel, and the flood control accuracy is improved. The scientificity and accuracy of flood disaster dangerous case identification, judgment and disposal decision making are improved, and the life and property safety of people is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of flood disaster risk, and specifically to a multimodal knowledge graph construction method and an expert system for flood disaster risk identification, judgment and disposal. Background Art

[0002] With the development of economy and society, the overall number of flood disasters is decreasing, and there are fewer and fewer experts with practical experience in identifying, judging and organizing flood disasters. However, flood disasters occur from time to time, and many front-line grassroots cadres lack corresponding experience in identifying and handling flood disasters. The present invention is based on the following technologies:

[0003] Knowledge graph is a technology for structured representation of knowledge, which can integrate and analyze a large amount of data and knowledge, thereby improving the efficiency and accuracy of information retrieval. With the continuous development of artificial intelligence technology, multimodal data has become the mainstream data form in the information age. Multimodal data refers to data that contains multiple media forms, such as text, images, audio and video. However, how to effectively process and analyze multimodal data remains a challenge. In this context, multimodal knowledge graph came into being. Multimodal knowledge graph is a technology that integrates information of different modalities into knowledge graph. It uniformly represents data of different media forms as nodes and edges in the graph, so that these data can be uniformly processed and analyzed. Compared with traditional knowledge graph, multimodal knowledge graph has richer semantic information and stronger representation ability.

[0004] In the field of water conservancy, the application of knowledge graphs has begun to take shape, but there are few studies on multimodal knowledge graphs for flood disaster risk identification and disposal. Establishing a multimodal knowledge graph for flood disaster risk identification, judgment and disposal can structure the knowledge of flood disaster risks. It has the advantages of fast retrieval speed and high knowledge correlation, and can meet the application needs of knowledge retrieval, knowledge matching, knowledge reasoning, etc., and improve the response and decision-making capabilities of flood control departments to flood disaster risks. Summary of the invention

[0005] In response to the above-mentioned problems, the present invention proposes a multimodal knowledge graph construction method and an expert system for identifying, judging and handling flood disaster risks.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A multimodal knowledge graph construction method for flood disaster risk identification, judgment and disposal, the method comprising the following steps:

[0008] Acquiring multimodal flood hazard data, wherein the multimodal flood hazard data includes two or more of text data, image data, video data, and audio data;

[0009] Extracting flood hazard knowledge based on the multimodal flood hazard data to obtain flood hazard knowledge triplets of each modality;

[0010] Multimodal flood hazard knowledge fusion and flood hazard knowledge expression are performed according to flood hazard knowledge triples to obtain a multimodal knowledge graph for flood hazards.

[0011] As a further technical solution of the present invention, the step of extracting flood hazard knowledge based on the multimodal flood hazard data to obtain flood hazard knowledge triples of each modality includes:

[0012] The text data is subjected to text semantic extraction through a text semantic extraction model to obtain knowledge triples of the text data;

[0013] Visual semantics extraction is performed on image data through a visual semantic extraction model to obtain knowledge triples of the image data.

[0014] As a further technical solution of the present invention, the steps of fusing multimodal flood hazard knowledge and expressing flood hazard knowledge according to flood hazard knowledge triples include:

[0015] The similarity between knowledge triples of different modalities is obtained through the semantic similarity algorithm;

[0016] When the similarity between knowledge triples of different modalities meets the preset conditions, multimodal flood hazard knowledge fusion is performed on the knowledge triples.

[0017] As a further technical solution of the present invention, the flood hazard knowledge triplet includes flood control entities, flood control relationships and hazard events, and the types of flood control entities include any one of the following or any combination thereof: water bodies, flood control projects, administrative regions, cultivated land, crops, houses, population, organizations (such as flood control headquarters, member units), rescue teams, rescue materials and equipment, etc.

[0018] As a further technical solution of the present invention, the flood control relationship includes any one of the following or any combination thereof: hazard classification, post-event summary, hazardous projects, signs of hazards, causes of hazards, in-event handling, possible causes and actions to be taken, etc.

[0019] As a further technical solution of the present invention, multimodal flood hazard data is obtained from one or more of flood control information systems, emergency plans, dispatch instruction records, inspection records, hazard notification documents, image data, water and rainfall historical databases, flood hazard charts, remote sensing data, and Internet data.

[0020] As a further technical solution of the present invention, the method also includes: storing and displaying a multimodal knowledge graph for flood disaster risk identification, judgment and disposal through a graph database management system.

[0021] Another object of the present invention is to provide a multimodal knowledge graph construction expert system for flood disaster risk identification, judgment and disposal, the expert system comprising:

[0022] A data acquisition module, used to acquire multimodal flood hazard data, wherein the multimodal flood hazard data includes two or more of text data, image data, video data, and audio data;

[0023] The knowledge extraction module is used to extract flood hazard knowledge based on multi-modal flood hazard data to obtain flood hazard knowledge triples of each modality;

[0024] The knowledge graph construction module is used to integrate and express multimodal flood hazard knowledge based on flood hazard knowledge triples, and obtain a multimodal knowledge graph for flood disaster hazard identification, judgment and disposal.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention acquires flood hazard data of different modalities to extract flood control knowledge, integrate multimodal flood hazard knowledge, and express flood hazard knowledge, thereby constructing a comprehensive multimodal knowledge graph dedicated to the field of flood disaster hazard prevention, which can provide reliable data guidance and support for flood control personnel, improve the scientificity and accuracy of flood disaster hazard identification, judgment and disposal decisions, and protect people's lives and property. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is the first flowchart of a multimodal knowledge graph construction method for flood disaster risk identification, judgment and disposal.

[0027] Figure 2 This is the second flow chart of the multimodal knowledge graph construction method for flood disaster risk identification, judgment and disposal.

[0028] Figure 3 This is an example diagram of the multimodal flood hazard knowledge graph to be constructed in an embodiment of the present invention.

[0029] Figure 4 Schematic diagram of the process of multimodal knowledge fusion in an embodiment of the present invention.

[0030] Figure 5 It is a schematic diagram of the process of fusing the external flood hazard knowledge triples with the flood hazard knowledge triples extracted by the present invention in an embodiment of the present invention.

[0031] Figure 6 This is a knowledge graph of a reservoir dam landslide hazard treatment plan in an embodiment of the present invention.

[0032] Figure 7 This is a schematic diagram of the multimodal knowledge graph provided in an embodiment of the present invention.

[0033] Figure 8 This is a precise flood prevention application diagram based on a multimodal flood hazard knowledge graph provided in an embodiment of the present invention.

[0034] Fig. 9 This is an example diagram of a personalized water conservancy project hazard file provided in an embodiment of the present invention.

[0035] Fig.10 A multimodal knowledge graph of flood control technologies related to reservoir landslide and levee breach hazards provided in an embodiment of the present invention.

[0036] Fig.11 This is a knowledge question and answer interface provided in an embodiment of the present invention.

[0037] Fig.12 The figure is a schematic diagram of the structure of an expert system for identifying, judging and handling flood disaster risks provided in an embodiment of the present invention.

[0038] Fig.13 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] like Figures 1 to 3 As shown, an embodiment of the present invention provides a multimodal knowledge graph construction method for flood disaster risk identification, judgment and disposal, and the method includes the following steps:

[0041] Step 1: Acquire multimodal flood hazard data, wherein the multimodal flood hazard data includes two or more of text data, image data, video data, and audio data;

[0042] Step 2: extracting flood hazard knowledge based on the multimodal flood hazard data to obtain flood hazard knowledge triples of each modality;

[0043] Step 3: Perform multimodal flood hazard knowledge fusion and flood hazard knowledge expression based on flood hazard knowledge triples to obtain a multimodal knowledge graph for flood hazards.

[0044] Step 4: Store and display the multimodal knowledge graph used for flood disaster risk identification, judgment and disposal through the graph database management system.

[0045] In this embodiment, step 1 can obtain multimodal flood control / flood hazard data from one or more of the water conservancy information system, command and dispatch records, post-flood summary documents, flood control and emergency training materials, image materials, water and rainfall databases, and Internet data. The multimodal flood control / flood hazard data includes data in various forms of expression, such as text form, image form, audio and video form, etc. The present invention aims to construct a multimodal knowledge graph that can fully cover various forms of flood control data. It is very important to clarify the data sources of various modal information for constructing a multimodal knowledge graph. Taking the pipe burst of a reservoir dam as an example, web page data related to "pipe burst of a reservoir dam" is obtained from the open network data, and empirical data is extracted from the flood control and emergency training materials. The structured data is stored in the Neo4j graph database, and the visual knowledge carriers such as pictures are saved in the form of files to form multimodal flood control data. After obtaining data of various modalities from different sources of flood control knowledge, data cleaning, data normalization, and manual revision can be performed to improve the accuracy of multimodal flood control data.

[0046] In step 2 of this embodiment, flood control entities can be defined in combination with the data characteristics of flood control / flood hazards and expert experience, and then knowledge extraction of flood control entities, flood control relationships and flood control / hazard events can be performed. The types of flood control entities can include any one of the following or any combination thereof: water bodies, flood control projects, administrative areas, cultivated land, crops, houses, population, organizations (such as flood control headquarters, member units), rescue teams, rescue materials, equipment, etc.; and flood control relationships can include any one of the following or any combination thereof: belonging to, hazard classification, post-event summary, hazardous projects, signs of hazard, causes of hazard, in-event disposal, possible causes, taking action, etc.

[0047] In this embodiment, knowledge extraction is a process of discovering implicit knowledge more deeply on the basis of information extraction, providing original materials and content for the construction of a multimodal flood risk knowledge graph. In the process of building a traditional knowledge graph, knowledge extraction is more targeted at implicit knowledge in the field, and the extraction process mainly relies on manual implementation. Based on the structure and modal characteristics of flood risk data, knowledge extraction mainly includes flood control entity recognition, flood control relationship extraction, and flood risk event extraction.

[0048] Specifically, refer to Figure 4, text semantics can be extracted from text data through a text semantic extraction model (such as the BERT model) to obtain knowledge triples of the text data; visual semantics can be extracted from image data through a visual semantic extraction model (such as the Faster R-CNN model) to obtain knowledge triples of the image data; for video data and audio data, they can be first converted into text data or image data, and then knowledge triples can be extracted.

[0049] As another preferred embodiment of the present invention, the step of fusing multimodal flood risk knowledge and expressing flood risk knowledge according to flood risk knowledge triples includes:

[0050] The similarity between knowledge triples of different modalities is obtained through the semantic similarity algorithm;

[0051] When the similarity between knowledge triples of different modalities meets the preset conditions, multimodal flood hazard knowledge fusion is performed on the knowledge triples.

[0052] In this embodiment, after extracting flood control / flood hazard knowledge based on multimodal flood hazard data, multimodal flood control entities, flood control relations, and flood hazard events are obtained. Step 3 can calculate the similarity between flood control entities through aggregation, clustering, representation learning and other methods, and calculate the similarity between flood control relations through edit distance, set similarity (Jaccard coefficient, Dice coefficient), vector similarity (cosine similarity, Euclidean distance) and other methods, and perform knowledge fusion on flood control entities or flood control relations whose similarity meets the preset range. Through knowledge fusion, knowledge from different knowledge sources is integrated, disambiguated, processed, reasoned and verified, updated and other steps of heterogeneous data under the same framework specification, so as to achieve the integration of data, information, methods, experience and human thoughts and form a high-quality knowledge base.

[0053] Specifically, the similarity between knowledge triples of different modalities can be obtained through a semantic similarity algorithm (such as a cosine similarity algorithm). When the similarity between knowledge triples of different modalities meets a preset condition (such as a preset threshold range), multimodal flood hazard knowledge fusion is performed on the knowledge triples.

[0054] As another preferred embodiment of the present invention, when constructing a multimodal flood risk knowledge graph, in addition to extracting knowledge from the original historical case data to obtain flood risk knowledge triples, the existing water conservancy information system, command and dispatch records, post-flood summary documents, flood prevention and emergency training materials, image materials, water and rainfall databases, and Internet data can also be used to supplement and expand knowledge to improve the richness of the constructed multimodal knowledge graph. However, due to the complexity of multi-source knowledge resources, entities or concepts from different data sources may point to the same object.

[0055] In this regard, multimodal flood risk knowledge fusion can be divided into multiple levels, such as entity fusion, data attribute fusion, and disambiguation of duplicated entities. Entity fusion is to address the problem that different data sources lack a unified identifier for the same entity, and to build semantic associations between entities in different data sources through fusion. For details, see Figure 5 . Attribute fusion is to solve the consistency problem of the attributes of the same entity in different data sources. The disambiguation of duplicated entities is the fusion of entities with the same name but different meanings, the same name but different meanings, and the same name from multiple sources in water conservancy objects (such as place names, water conservancy projects, organizations, etc.), which is mainly matched based on the entity structure and the attribute characteristics of the entity. When processing the disambiguation of water conservancy entities, all data nodes are projected into a global unified representation space, and a local representation function is constructed for each candidate set to measure the similarity between two entity nodes. Then the nodes are clustered based on the similarity, and the candidate set is split according to the clustering results to obtain the final data fusion result. Taking the fusion of entities as an example, if the entity "Yangtze River Water Conservancy Commission of the Ministry of Water Resources" is extracted from the encyclopedia entry, and the institutional data imported from the industry database contains the "Yangtze River Commission" entity, these two entity names are descriptions of the same entity, but because of different data sources, the two need to be fused. First, the similarity of the two strings is calculated using the edit distance metric, then the similarity of the related attributes is analyzed based on the attribute information of the two entities, and finally the graph structure similarity is calculated based on the graph relationship between the two entities (such as the relationship between superiors and subordinates). By comprehensively analyzing the above three similarities, if it is greater than the set threshold, the two entities are merged.

[0056] The data after knowledge fusion is then used for knowledge expression, and ontology technology can be used to semantically express multimodal knowledge. The terms in the water conservancy field are described using network ontology language, and the flood control entities (classes), flood control relationships (attributes), and flood hazard event classes (instances) of flood hazard-related knowledge are clearly defined. In Protege, OntoGraf is used to draw class and instance relationships to construct a knowledge graph for flood hazard events, and seven classes are listed, including "hazard events", "water conservancy projects", "flood hazard categories", "scenario", "cause of danger", "hazard hazard", and "hazard disposal plan". The association between the instance "1995 Hunan Changde Yuangui Reservoir Earth Dam Landslide Rescue" and the "Yuangui Reservoir" instance is established through attributes, see Figure 6 .

[0057] It should be noted that after knowledge extraction, knowledge fusion and knowledge expression, the unstructured original multimodal flood hazard data forms a structured knowledge expression form of flood hazard knowledge triples of <entity, relationship, entity>, and obtains a multimodal knowledge graph. Neo4j is a high-performance graph database management system that can represent data through its graph data model, store entities and relationships in flood hazard knowledge triples through nodes and edges, and display multimodal knowledge graphs through Neo4j's visualization tools to facilitate knowledge query and management.

[0058] The multimodal knowledge graph construction method for flood disaster hazards provided by the present invention constructs a multimodal knowledge graph of information in different forms of expression covering hazard warning, hazard identification, hazard hazards, hazard disposal, etc. through a four-stage method of "knowledge source determination-knowledge extraction-knowledge fusion and expression-knowledge storage and display". It makes full use of information in various different modes such as images, expressions, videos, audio, etc. in addition to text, and describes objects through different structural features and phenotypes, which can form a more complete and accurate representation and evaluation. The flood season is characterized by frequent rainfall and abundant rainfall, which can easily cause flood disaster hazards and cause serious economic losses and casualties. The multimodal knowledge graph can provide flood control personnel with professional analysis and judgment of the hazard situation and expert advice on hazard disposal measures, and assist flood control personnel in preventing, identifying, and disposing of hazards.

[0059] It should be noted that the data of the multimodal knowledge graph can be updated regularly to improve the integrity and data reliability of the multimodal knowledge graph, thereby improving the quality of flood prevention personnel's identification, judgment and disposal decisions of flood disaster risks based on the multimodal knowledge graph.

[0060] As another preferred embodiment of the present invention, this embodiment provides a flood disaster risk management decision support system based on a multimodal knowledge graph, including:

[0061] A multimodal knowledge graph construction module, used for constructing a multimodal knowledge graph according to the multimodal knowledge graph construction method for flood disaster risk identification, judgment and disposal;

[0062] A flood hazard knowledge search module is used to feed back corresponding flood hazard knowledge to the user based on the flood hazard query instruction input by the user and the multimodal knowledge graph, wherein the flood hazard knowledge includes flood prevention knowledge;

[0063] The flood hazard question and answer module is used to provide users with feedback on flood disaster hazard identification, judgment and disposal plan recommendations based on the flood hazard question information input by the user and the multimodal knowledge graph.

[0064] It should be noted that the flood hazard problem may be a flood disaster hazard event related to a water conservancy project, and the hazard identification, judgment and disposal plan recommendation information may be a flood control scheduling plan and / or hazard disposal measures related to the flood disaster hazard event; the flood disaster hazard query instruction may include one or more keywords, codes, numbers, text files, audio files, video files, etc.

[0065] The present invention is based on a multimodal knowledge graph construction method for flood disaster risk identification, judgment and disposal. It utilizes water conservancy information systems, command and dispatch records, post-flood summary documents, flood prevention and emergency training materials, image materials, water and rainfall databases, and Internet data. Through the steps of knowledge extraction, knowledge fusion, knowledge expression, and knowledge storage, a multimodal flood disaster risk identification, judgment, and disposal knowledge graph with 6292 entities and 33025 relationships is constructed. Figure 7 The schematic diagram of the multimodal flood control / flood disaster risk identification, judgment and disposal knowledge graph is shown. The flood control entities include 9 types, such as water conservancy projects, risk events, risk categories, risk causes, water conditions, rainfall conditions, engineering conditions, risk hazards, and risk disposal plans. The flood control relations include 12 types, such as belonging to, risk classification, post-event summary, risk engineering, risk signs, risk causes, in-event disposal, possible causes, and actions. Table 1 counts the number of flood control entities and flood control relations in the multimodal flood risk knowledge graph.

[0066] Table 1 - Statistics of flood control entities and flood control relationships in the multimodal flood risk knowledge graph

[0067] Entity Classification quantity Relationship Classification quantity Dangerous incidents 107 inherit 734 Water Conservancy Project 107 Hazard Classification 28 Flood hazard category 23 Postmortem 107 Dangerous situation handling plan 107 Dangerous Project 107 Cause of accident 326 Signs of danger 428 scene 107 Cause of danger 344 Water conditions 209 In-process handling 508 Rainy mood 307 May cause 652 Work 439 Take Action 783

[0068] Based on the multimodal flood disaster risk identification, judgment and disposal knowledge graph constructed above, an expert system for accurate flood disaster risk identification, judgment and disposal can be built. Its main applications include building personalized flood risk archives for water conservancy projects, semantic search for flood risks, instant question and answer on flood risks, and personalized flood disaster risk disposal plan decision-making, such as Figure 8 shown.

[0069] Constructing personalized flood and waterlogging risk archives for water conservancy projects: Collect flood and waterlogging risk data for water conservancy projects from various sources, including water conservancy information systems, command and dispatch records, post-flood summary documents, flood control and emergency training materials, image materials, water and rain databases, Internet data, etc. These data can be in the form of text, images, videos, or other forms, forming a multimodal flood control / flood and waterlogging risk identification, judgment, and disposal plan data set. Preprocess the collected data to ensure the accuracy, completeness, and consistency of the data. Then, integrate and transform the multimodal flood and waterlogging risk data so that it can be uniformly represented in the knowledge graph. Using the technology of knowledge graphs, different types of data can be fused, connected, and associated. For example, the basic information, risk overview, risk category, disposal plan, risk cause, (water and rain engineering) scenario, etc. of the water conservancy project in danger can be connected to form a complete multimodal flood and waterlogging risk knowledge graph. Add attributes and labels to each node and edge so that the knowledge graph can express rich information. For the sign nodes of water conservancy projects in danger, attributes such as water level, flow, rainfall, wind speed, water surface status (bubbles, turbidity), and water seepage can also be added. The health status of water conservancy projects in danger may change over time, so the personalized flood hazard archives of water conservancy projects will also be continuously updated. By continuously adding new data and information to the multimodal flood hazard knowledge graph, the accuracy and practicality of the archives can be maintained. Ultimately, a personalized flood hazard archive for water conservancy projects in danger can be formed, providing a solid foundation for on-site flood control decisions.

[0070] Fig. 9 The document shows some personalized flood control / flood disaster risk identification, judgment and disposal archives for water conservancy projects. With risk events as the center, the knowledge base of flood disaster risk constructed with relationships such as belonging, risk classification, post-event summary, risk project, risk signs, risk causes, in-event disposal, possible causes, and actions taken as edges is connected to form a multi-modal flood disaster risk identification, judgment and disposal knowledge map that can be directly applied to grassroots flood control.

[0071] About semantic search for flood hazards: Semantic search for flood hazards refers to the process of using a multimodal flood hazard knowledge graph as a database, defining keywords and terms, and using query languages ​​or tools to retrieve information related to specific flood disaster hazards in the multimodal flood hazard knowledge graph. Through semantic search, flood prevention personnel can quickly find nodes and edges related to specific hazards, hazards, or disposal solutions, and obtain comprehensive and accurate flood disaster hazard identification, judgment, and disposal solution decision support. In addition, semantic search can also apply semantic reasoning technology to expand the search scope, find other nodes and information related to the search target, and provide flood prevention personnel with more comprehensive flood disaster hazard identification, judgment, and disposal support.

[0072] At the same time, by using semantic search of flood hazards, flood prevention personnel can easily conduct knowledge retrieval and skills training. Figure 8 The application of accurate flood control / flood hazard defense based on multimodal flood system hazard knowledge graph is demonstrated. If new flood control / flood prevention personnel want to know the treatment plan for reservoir dam landslide, they can search for keywords such as "reservoir" and "landslide" to quickly obtain guidance on flood control treatment plans for the hazard. Fig.10 A multimodal flood control knowledge graph of flood control technologies related to reservoir landslide and levee breach hazards.

[0073] Instant Q&A on flood hazards: The flood hazard Q&A module uses information and data in the multimodal flood hazard knowledge graph to provide flood prevention personnel with real-time, accurate flood disaster hazard identification, judgment questions and guidance. Flood prevention personnel can enter keywords or problem descriptions, and the module will automatically use the multimodal flood hazard knowledge graph for semantic search and give corresponding flood disaster hazard identification, judgment suggestions and solutions. Fig.11 The question-and-answer interface built based on the multimodal flood hazard knowledge graph is displayed. The left side of the interface contains modules such as data upload, hazard archive record, flood hazard semantic search, and knowledge question-and-answer. The main body of the interface is a question-and-answer module, with question input at the top, answers and related knowledge graphs at the bottom, and knowledge recommendations at the bottom. This module can integrate data collection, knowledge question-and-answer, and intelligent decision-making, and achieve good application results for flood prevention and disaster reduction practices in water conservancy projects.

[0074] Regarding personalized flood control / flood disaster hazard disposal plan decision-making: Based on the multimodal flood control / flood disaster hazard identification, judgment and disposal knowledge graph, it is possible to automatically diagnose by constructing a personalized flood control / flood disaster hazard disposal plan decision-making module, and provide flood disaster hazard identification, judgment and disposal plans based on hazard signs, flood control situation / water and rain conditions, to help flood control / flood prevention personnel make correct flood disaster hazard identification and judgment in a timely manner, and improve the quality of flood control work. The personalized flood control / flood disaster hazard disposal plan decision-making module based on the multimodal flood control / flood disaster hazard identification, judgment and disposal knowledge graph can complete the decision support process through the reasoning engine. When the user enters the hazard scenario, the reasoning engine will automatically feedback the hazard results or recommend hazard disposal plans based on the multimodal flood disaster hazard identification, judgment and disposal knowledge graph and the user's input.

[0075] Reference Fig.12 Another object of the present invention is to provide an expert system for identifying, judging and handling flood disasters, the expert system comprising:

[0076] A data acquisition module, used to acquire multimodal flood hazard data, wherein the multimodal flood hazard data includes two or more of text data, image data, video data, and audio data, and in this embodiment may be multimodal flood hazard data of a water conservancy project;

[0077] The knowledge extraction module is used to extract flood hazard knowledge based on multi-modal flood hazard data to obtain flood hazard knowledge triples of each modality;

[0078] The knowledge graph construction module is used to integrate and express multimodal flood hazard knowledge based on flood hazard knowledge triples, and obtain a multimodal knowledge graph for flood disaster hazard identification, judgment and disposal.

[0079] Fig.13 An example of a physical structure diagram of an electronic device is shown, and the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a method for constructing a multimodal knowledge graph for flood disaster risk identification, judgment, and disposal.

[0080] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0081] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal knowledge graph construction method for flood disaster risk identification, judgment and disposal provided by the above-mentioned methods.

[0082] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal knowledge graph construction method for flood disaster risk identification, judgment and disposal provided by the above-mentioned methods.

[0083] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0084] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, a mobile hard disk, a USB flash drive, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0085] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0086] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multimodal knowledge graph construction method for flood disaster risk identification, judgment and disposal, characterized in that: The method comprises the following steps: Acquiring multimodal flood hazard data, wherein the multimodal flood hazard data includes two or more of text data, image data, video data, and audio data; Extracting flood hazard knowledge based on the multimodal flood hazard data to obtain flood hazard knowledge triplets of each modality; Multimodal flood hazard knowledge fusion and flood hazard knowledge expression are performed according to flood hazard knowledge triples to obtain a multimodal knowledge graph for flood hazards.

2. The multimodal knowledge graph construction method for flood disaster risk identification, judgment and disposal according to claim 1 is characterized in that: The step of extracting flood hazard knowledge according to the multi-modal flood hazard data to obtain flood hazard knowledge triples of each modality comprises: The text data is subjected to text semantic extraction through a text semantic extraction model to obtain knowledge triples of the text data; Visual semantics extraction is performed on image data through a visual semantic extraction model to obtain knowledge triples of the image data.

3. The multimodal knowledge graph construction method for flood disaster risk identification, judgment and disposal according to claim 2 is characterized in that: The steps of fusing multimodal flood risk knowledge and expressing flood risk knowledge according to flood risk knowledge triples include: The similarity between knowledge triples of different modalities is obtained through the semantic similarity algorithm; When the similarity between knowledge triples of different modalities meets the preset conditions, multimodal flood hazard knowledge fusion is performed on the knowledge triples.

4. The method for constructing a multimodal knowledge graph for flood disaster risk identification, judgment and disposal according to any one of claims 1 to 3, characterized in that: The flood hazard knowledge triplet includes flood control entities, flood control relationships and hazard events. The types of flood control entities include any of the following or any combination of them: water bodies, flood control projects, administrative regions, cultivated land, crops, houses, population, organizations, rescue teams, rescue materials and equipment, etc.

5. The multimodal knowledge graph construction method for flood disaster risk identification, judgment and disposal according to claim 4 is characterized in that: The flood control relationship includes any one of the following or any combination thereof: hazard classification, post-event summary, hazardous projects, signs of hazard, causes of hazard, in-event handling, possible causes and actions taken.

6. The method for constructing a multimodal knowledge graph for flood disaster risk identification, judgment and disposal according to any one of claims 1 to 3, characterized in that: Multimodal flood hazard data are obtained from one or more of flood control information systems, emergency plans, dispatch instruction records, inspection records, hazard notification documents, image data, water and rainfall historical databases, flood hazard charts, remote sensing data, and Internet data.

7. The method for constructing a multimodal knowledge graph for flood disaster risk identification, judgment and disposal according to any one of claims 1 to 3, characterized in that: The method also includes: storing and displaying a multimodal knowledge graph for flood disaster risk identification, judgment and disposal through a graph database management system.

8. A multimodal knowledge graph expert system for flood disaster risk identification, judgment and disposal, characterized by: The expert system comprises: A data acquisition module, used to acquire multimodal flood hazard data, wherein the multimodal flood hazard data includes two or more of text data, image data, video data, and audio data; The knowledge extraction module is used to extract flood hazard knowledge based on multi-modal flood hazard data to obtain flood hazard knowledge triples of each modality; The knowledge graph construction module is used to integrate and express multimodal flood hazard knowledge based on flood hazard knowledge triples, and obtain a multimodal knowledge graph for flood disaster hazard identification, judgment and disposal.

Images