Multi-modal landslide event entity extraction method

Abstract

The invention provides a multi-modal landslide event entity extraction method, which comprises the following steps of: firstly, constructing a landslide entity expression model taking'feature expression and relation expression 'as a core, and then, providing an instruction prompt driven cross-platform multi-modal landslide event data set construction method. And constructing a multi-modal landslide event data set. Then, a multi-modal landslide event entity extraction model based on prompt engineering is constructed, the model is based on the prompt engineering concept, deep understanding of the landslide event is achieved by fusing an instruction prompt statement set of structured knowledge, and finally comprehensive description of a landslide event entity is conducted through a classification fusion strategy. According to the invention, the problems of low landslide event data processing efficiency, low entity extraction accuracy, difficulty in multi-modal data fusion and the like in the prior art are effectively solved, and powerful support is provided for landslide disaster early warning and emergency management.

Description

Technical Field

[0001] The present invention relates to the technical field of information management, and more specifically, to a multi-modal landslide event entity extraction method. Background Art

[0002] Multimodal data provides a more comprehensive information perspective for landslide event management, in which each modal data contributes its unique perspective and depth. For example, text data records in detail the occurrence process, historical background and related geographical information of landslide events, while image data intuitively shows the on-site conditions, terrain changes and disaster impact range of landslides. The complementarity and mutual verification of such multimodal data not only enrich the information dimension, but also enhance the credibility of the data. However, in the face of increasing multimodal landslide event information, current research still has problems such as complex conceptual models, low data processing efficiency, low entity extraction accuracy, and difficulty in multimodal data fusion, which brings many challenges to landslide event management. These challenges are mainly concentrated in two aspects: one is how to effectively organize, express and associate various complex landslide event information; the other is how to efficiently extract effective landslide event information from multimodal data and realize the effective fusion of different modal data extraction results.

[0003] In recent years, large language models have shown remarkable capabilities in processing multimodal data, but their performance in specific fields such as multimodal landslide event entity extraction still has certain limitations. This is mainly because general large language models lack deep optimization and customized processing of data in specific fields. In the task of multimodal landslide event entity extraction, the model not only needs to be able to recognize and understand the description of landslide events in text data, but also must be able to associate and parse the corresponding image content. In this context, the application of prompt engineering is particularly critical. Through carefully designed prompt instructions, prompt engineering can effectively guide large language models to process specific tasks more intensively, thereby significantly improving the accuracy and efficiency of multimodal data extraction. These prompt instructions are like precise navigation, helping the model to locate and extract key entity information in a complex data environment. Therefore, developing a special set of instruction prompts for the field of landslide disasters is of great significance to improving the level of landslide disaster prevention and management. Summary of the invention

[0004] Aiming at the technical problems existing in the prior art, the present invention proposes a multi-modal landslide event entity extraction method based on a landslide field knowledge reference system by analyzing the characteristics of landslide event data.

[0005] The present invention provides a multi-modal landslide event entity extraction method, comprising:

[0006] Step 1, establishing a landslide event entity and entity relationship expression framework to establish a landslide entity expression model, wherein the landslide event entity and entity relationship expression framework includes a feature expression system of the landslide event entity and a relationship expression system of the landslide time entity;

[0007] Step 2, based on the established landslide entity expression model, using automatic crawler technology to obtain multimodal landslide event data from multiple channels, and preprocessing and annotating the multimodal landslide event data based on instruction prompt driving, wherein the multimodal landslide event data includes landslide event text data and landslide event image data;

[0008] Step 3, constructing a multimodal landslide event entity extraction model based on prompt engineering, extracting the landslide event text data and the landslide event image data respectively, and obtaining a text data extraction result and an image data extraction result;

[0009] Step 4: fuse the text data extraction result and the image data extraction result to obtain a landslide event entity extraction fusion result.

[0010] The multi-modal landslide event entity extraction method provided by the present invention has the following beneficial effects:

[0011] Aiming at the complexity and multi-source of landslide event data, the present invention constructs a landslide entity expression framework with "feature expression and relationship expression" as the core. This expression framework provides a theoretical basis for landslide event entity recognition and a new idea for the organization, association and management of landslide event data.

[0012] A command prompt-driven method for constructing a cross-platform multimodal landslide event dataset is proposed. Based on the automatic crawling of cross-platform multimodal landslide event data, the present invention develops a dedicated command prompt to automatically convert the fuzzy time and fuzzy space representations in the data, and adopts an efficient data annotation process combined with multi-person online annotation and expert review to construct a high-quality multimodal landslide event dataset. This provides an effective way to construct a multimodal landslide event dataset.

[0013] A multimodal landslide event entity extraction model based on prompt engineering is constructed. Based on the prompt engineering concept, this model proposes a framework for efficiently extracting landslide event entities from multimodal data, and designs a set of landslide event extraction instruction prompt statements specifically for multimodal data. The block extraction strategy, result verification and correction mechanism, and classification fusion strategy are used to achieve accurate quantification and comprehensive description of landslide event entities, providing new ideas and methods for extracting landslide events from multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flow chart of a multi-modal landslide event entity extraction method provided by the present invention;

[0015] Figure 2 The whole framework structure diagram of a multi-modal landslide event entity extraction provided by the present invention;

[0016] Figure 3 Label the data with a flow chart;

[0017] Figure 4 The architecture diagram of the multimodal landslide event entity extraction model based on prompt engineering. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be arbitrarily combined with each other to form a feasible technical solution. This combination is not subject to the constraints of the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0019] Based on the defects in the background technology, the basic idea of ​​the present invention is: first, to construct a landslide event entity expression framework with "feature expression and relationship expression" as the core. Secondly, the automatic crawler technology and the special instruction prompt system are used to realize the automatic collection and standardized processing of cross-platform multimodal data, providing a high-quality data foundation for data annotation and entity extraction. Finally, a multimodal entity extraction model based on prompt engineering is constructed. The model extracts landslide event entities from text and images through modular design and classification fusion strategy, and improves the accuracy and robustness of the model through result verification and correction mechanism. Finally, the classification fusion strategy is used to realize the accurate quantification and comprehensive description of landslide event entities.

[0020] Figure 1 A flow chart of a multi-modal landslide event entity extraction method provided by the present invention is as follows: Figure 1 and Figure 2 As shown, the method includes:

[0021] Step 1: Establish a landslide event entity and entity relationship expression framework to establish a landslide entity expression model. The landslide event entity and entity relationship expression framework includes a feature expression system of the landslide event entity and a relationship expression system of the landslide time entity.

[0022] It is understandable that the landslide event entity and entity relationship expression framework is established, and a landslide entity expression model with "feature expression and relationship expression" as the core is constructed to target the complexity and multi-source heterogeneity of landslide event data. This landslide entity expression model can not only accurately capture and express the diversity and dynamics of landslide events, but also reveal the internal connection and interaction mechanism between different landslide event entities, providing a powerful tool for in-depth understanding of the occurrence and development of landslide events.

[0023] The landslide event entity and entity relationship expression framework includes the feature expression system of the landslide event entity and the relationship expression system of the landslide event entity.

[0024] Among them, the establishment of a landslide event entity feature expression system includes:

[0025] The semantic description is defined by the landslide event name (E name ) and landslide event code (E ID )composition;

[0026] The spatial location is defined by longitude (E longitude ), latitude (E latitude ), administrative divisions (E admiArea ), detailed address (E address ) and administrative area code (E admiID )composition;

[0027] Define the time evolution by the occurrence time (E time ) and time evolution (E evolution )composition;

[0028] The attribute characteristics are defined by the landslide grade (E level ), landslide type (E type ), landslide hazard factor (E factors ), landslide disaster-prone environment (E environment ), landslide hazard-bearing body (E suffer ), Pre-disaster warning (E warning ), Disaster Rescue (E rescue ), post-disaster reconstruction (E reconstruction ), landslide geometric characteristics (E geometry ) and dynamic characteristics of landslides (E dynamic )composition.

[0029] Specifically, a characteristic expression system of landslide event entities is established from four dimensions: semantic description, spatial location, attribute characteristics and time evolution, and the component elements and expression forms are given.

[0030] 1) The semantic features of landslide event entities are the precise description and definition of landslide events, focusing on the description of the essential information of landslide events. The formal expression of the semantic features of landslide event entities is as follows:

[0031] E semantics ={E name ,E ID (1)

[0032] Among them, E semantics represents the landslide semantic description entity, E name Indicates the name of the landslide event, E ID The unique identification code of the landslide event.

[0033] 2) Spatial position is the basis for expressing the spatial state of the landslide event entity. The spatial position of a landslide event reflects the specific spatial area information of a landslide event, and is the most basic and core description of the spatial characteristics of a landslide event. The formal expression of the spatial position of a landslide event entity is as follows:

[0034] E space ={E longitude ,E latitude ,E admiArea ,E address ,E admiID} (2)

[0035] Among them, E space represents the spatial location entity of the landslide event, E longitude Indicates the longitude of the landslide event; E latitude Indicates the latitude of the landslide event; E admiArea The administrative area of ​​the landslide event mainly refers to: the smallest external administrative area that can completely include the spatial location of the landslide event; E address Indicates the detailed address of the landslide event to the village or community; E admiID Represents the administrative region code, which is a standardized expression of administrative divisions.

[0036] 3) Time evolution mainly focuses on the specific occurrence time and evolution process of landslide events. The formal expression of the time evolution of landslide events is as follows:

[0037] E Time ={E OCtime ,E evolution1 ,E evolution2 ,E evolution3 …} (3)

[0038] Among them, E Time represents the time evolution entity of the landslide event, E OCtime Indicates the time of occurrence of landslide event; E evolution1Indicates the first evolution time of the landslide event; E evolution2 represents the second evolution time of the landslide event; E evolution3 Indicates the third evolution time of the landslide event.

[0039] 4) Attribute characteristics describe the essential attributes, occurrence conditions, impact range, emergency response, and recovery process of landslide events. The formal expression of the attribute characteristics of landslide events is as follows:

[0040]

[0041] Among them, E level Indicates the landslide grade; E type Indicates the landslide type; E factors Indicates landslide hazard factor; E environment Indicates landslide disaster-prone environment; E suffer Indicates the landslide hazard-bearing body; E warning Indicates pre-disaster warning; E rescue Indicates disaster relief; E reconstruction Indicates post-disaster reconstruction; E geometry Represents the geometric characteristics of the landslide; E dynamic Represents the dynamic characteristics of landslide.

[0042] Establish a landslide event entity relationship expression system, including:

[0043] Definition of disaster relationship: Disaster relationships are established based on the cause-effect chain and interaction mechanism of disaster occurrence. The relationship types include "primary disaster", "concurrent disaster" and "secondary disaster".

[0044] The definition of spatial association relationship is established based on topological hierarchy relationship and relative position of orientation. The relationship types include "subject to" and "occurs in". The reference knowledge is the administrative division hierarchy set.

[0045] The time correlation relationship is defined based on the chronological order of landslide events. The relationship types include "same year", "same month", "earlier than" and "later than".

[0046] The definition of attribute association relationships is based on the similarity relationships in the landslide event attributes. The relationship types include "similar disaster-causing factors", "similar disaster-prone environments", "similar disaster-bearing bodies", "same landslide levels", "same landslide types", "similar rescue forms during disasters", "similar pre-disaster warnings" and "similar post-disaster reconstruction".

[0047] Specifically, a relational expression system of landslide event entities is established from four aspects: disaster correlation, spatial correlation, temporal correlation and attribute correlation, and relationship descriptions and association explanations are made.

[0048] 1) Disaster correlation: Landslide disasters usually do not occur in isolation, and their development is closely linked to other types of disasters, showing a certain linkage effect. For example, earthquake-induced landslides may further lead to road closures or river obstructions, resulting in the formation of barrier lakes. Considering the non-independent nature of landslide disasters, this paper analyzes landslide events based on the framework of comprehensive natural disaster management, and explores the complex connection and mechanism of action between landslide disasters and other types of disaster events. Disaster relationships can be defined as:

[0049] R disaster = {R PriDisaster ,R ConcDisaster ,R SeconDisaster} (5)

[0050] Where R PriDisaster Represents primary disasters, such as landslide disasters. Primary disasters may be rainstorm disasters; R ConcDisaster Indicates concurrent disasters, such as mudslides and debris flows. SeconDisaster It indicates secondary disasters, such as the secondary disaster of a landslide may be a barrier lake.

[0051] 2) Spatial association relationship is the location association generated by the distribution of landslide event entities in space. The spatial association relationship is expressed by the hierarchical inclusion in the topological relationship and the relative position in the orientation. The relationship types include "subordinate", "occurred in" and "** kilometers away". The reference knowledge is the administrative division level set. Specifically, there is a "subordinate" relationship between the nodes of the administrative division set, and there is an "occurred in" relationship between the landslide event and the specific region. The landslide spatial relationship can be expressed as:

[0052] R space = {R topology ,R distance ,R direction} (6)

[0053] Among them, R space is the spatial relationship of landslide, R topology is the spatial topological relationship; R direction Indicates the orientation relationship, R direction Indicates the distance between landslide event entities.

[0054] 3) Time correlation relationship: Time correlation relationship is established based on the time sequence of landslide events. The relationship types include "same year", "same month", "earlier than" and "later than". The landslide time relationship can be defined as:

[0055] R time = {R year ,R month} (7)

[0056] Among them, Rtime is the landslide time relationship; R year R is the time sequence of the year; month Indicates the time sequence of months.

[0057] 4) The attribute association relationship is established based on the similarity relationship among the landslide event attributes. The attribute relationship can be defined as:

[0058]

[0059] Among them, R level Represents the similarity relationship of landslide grade; R type Represents the similarity relationship of landslide types; R factors Represents the similarity relationship between landslide hazard factors; R environment Represents the similarity relationship of landslide disaster-prone environment; R suffer Represents the similarity relationship of landslide hazard-bearing bodies; R warning Represents the similarity relationship of pre-disaster warning; R rescue Indicates disaster relief; R reconstruction Represents the similarity relationship of post-disaster reconstruction; R geometry Represents the similarity relationship of landslide geometric characteristics; R dynamic Represents the similarity relationship of landslide dynamic characteristics.

[0060] Step 2: Based on the established landslide entity expression model, automatic crawler technology is used to obtain multimodal landslide event data from multiple channels, and the multimodal landslide event data is preprocessed and labeled based on instruction prompt drive, wherein the multimodal landslide event data includes landslide event text data and landslide event image data.

[0061] Exemplarily, the preprocessing and labeling of the multimodal landslide event data based on the instruction prompt drive includes:

[0062] Preprocessing the landslide event text data, wherein the preprocessing includes removing irrelevant information, segmenting words, and removing stop words;

[0063] For the fuzzy time words and fuzzy space words in the landslide event text data, a first instruction prompt sentence is set, wherein the first instruction prompt sentence instructs the large language model to understand the task objectives, processing methods and output formats, and the first instruction prompt sentence includes a task description, a conversion example and text data to be processed, and guides the large language model to convert the fuzzy time words and fuzzy space words in the landslide event text data based on context information;

[0064] Performing image noise reduction, format conversion and size adjustment on the landslide event image data;

[0065] Entity types and annotation rules are defined, and the processed landslide event text data and landslide event image data are annotated based on the defined entity types and annotation rules.

[0066] It is understandable that based on the landslide event entity expression framework defined in step 1, the present invention develops a dedicated instruction prompt (hereinafter referred to as the first instruction prompt statement) on the basis of using automatic crawler technology to realize the automatic collection of cross-platform multimodal landslide event data, and combines the large language model to realize the automatic standardized conversion of fuzzy time and fuzzy space expressions in the data. This technology can solve the problem of ambiguity in time and space references of data, and lay a solid data foundation for subsequent data annotation and entity extraction tasks. In addition, the present invention adopts a set of efficient data annotation processes, combined with a multi-person online annotation platform and a domain expert review mechanism, to construct a high-quality data annotation set. The specific content of this step is as follows:

[0067] (1) Automated acquisition of multi-modal landslide data across platforms:

[0068] The present invention adopts automatic crawler technology to obtain multi-modal landslide event data from multiple channels, including:

[0069] 1) Obtain reliable text data and image data of landslide events from the website, including text description, geographic information, time information, landslide site pictures, etc.

[0070] 2) Obtain scientific research and monitoring data related to landslide events from scientific research data platforms such as the Resources and Environmental Science Data Platform, including landslide mechanism analysis, monitoring data, etc.

[0071] 3) Obtain academic papers related to landslide events from academic databases and obtain professional research and analysis of landslide events.

[0072] 4) Obtain news reports and news photos of landslide events from news media and obtain real-time landslide event information.

[0073] 5) Obtain public discussions and comments on landslide events from social media platforms, and obtain text data and image data of landslide events from different angles and locations.

[0074] (2) Landslide event data preprocessing driven by command prompts:

[0075] For the collected multimodal data, the present invention performs data preprocessing operations, including:

[0076] 1) Preprocessing of landslide event text data: remove irrelevant information, perform word segmentation, remove stop words, etc. to improve the quality and usability of text data. At the same time, use the large language model to automatically convert fuzzy time words and fuzzy space words. The steps for automatically converting fuzzy time words and fuzzy space words are as follows:

[0077] Construction of instruction prompts: The present invention designs a dedicated first instruction prompt sentence for the task of replacing fuzzy time words and fuzzy space words in landslide event text data, which clearly instructs the large language model to understand the task objectives, processing methods and output formats. The first instruction prompt sentence contains task description, conversion examples, text to be processed, etc., and guides the large language model to convert fuzzy time words and fuzzy space words based on context information.

[0078] Replacement based on context information: The large language model analyzes the context information of the text data according to the first instruction prompt and its own semantic understanding ability, identifies fuzzy time words and fuzzy space words, and combines the knowledge of landslide field to convert the fuzzy event words and fuzzy space words in the text data into accurate year, month, day, hour, minute, second format and province, city, county, district, town location representation. 2) Preprocessing of landslide event image data:

[0079] Image processing technology is used to perform operations such as image noise reduction, format conversion, and size adjustment to improve the quality and usability of image data.

[0080] (3) Construction of high-quality annotated dataset:

[0081] After the landslide event text data and image data are preprocessed, the landslide event entities in the text data and image data are annotated to obtain the landslide event dataset. Figure 3 As shown: Before annotating document data, you need to define entity types and annotation rules, then create annotation tasks and distribute them to teams and authors to enable multiple people to annotate online. Finally, check the annotation quality. If the annotation is qualified, export the annotation results and end the process. If the annotation is unqualified, re-annotate. The specific steps are as follows:

[0082] 1) Multi-person online annotation: Use Label Studio multi-person online annotation platform to perform detailed annotation of text data, realize multi-person online joint annotation, and improve annotation efficiency and quality.

[0083] 2) Quality control of data annotation: The present invention adopts strict data annotation quality control measures, which mainly include the following two parts:

[0084] Cross-validation: Cross-validate the annotation results to ensure their accuracy and consistency.

[0085] Expert review: Domain experts review the annotation results to ensure their accuracy and professionalism.

[0086] Step 3: construct a multimodal landslide event entity extraction model based on prompt engineering, extract the landslide event text data and the landslide event image data respectively, and obtain text data extraction results and image data extraction results.

[0087] Exemplarily, the multimodal landslide event entity extraction model includes a text information extraction module, a picture information extraction module and a result fusion module. The step 3 constructs a multimodal landslide event entity extraction model based on the prompt engineering, extracts the landslide event text data and the landslide event picture data respectively, and obtains text data extraction results and picture data extraction results, including:

[0088] A second instruction prompt sentence for extracting landslide event entities is constructed based on the text information extraction model, and according to the second instruction prompt sentence, landslide event entities are extracted from landslide event text data based on a large language model to obtain a text data extraction result;

[0089] Constructing a third instruction prompt sentence for extracting landslide event entities based on the picture information extraction model, and extracting landslide event entities from landslide event picture data based on the large language model according to the third instruction prompt sentence to obtain picture data extraction results;

[0090] The text data extraction result and the image data extraction result are fused based on the result fusion module.

[0091] Specifically, the processed landslide event dataset is obtained through step 2, and a multimodal landslide event entity extraction model based on prompt engineering is constructed to extract landslide event entities from the landslide event dataset. Among them, the multimodal landslide event entity extraction model is based on the prompt engineering concept, proposes a framework for efficiently extracting landslide event entities from multimodal data, and designs a set of landslide event extraction instruction prompt statements dedicated to multimodal data. The architecture of the multimodal landslide event entity extraction model is as follows: Figure 4 The model mainly consists of three core modules: text information extraction module, image information extraction module and result fusion module. The following are the descriptions of these three models:

[0092] (1) Text information extraction module:

[0093] Construct a second instruction prompt sentence dedicated to landslide event entity extraction to efficiently extract landslide event entities from the large language model.

[0094] 1) The second instruction prompt statement is generated:

[0095] The second instruction prompt statement of the text information extraction module mainly consists of four parts:

[0096] Landslide structured knowledge: Landslide structured knowledge provides relevant background knowledge about landslide events, aiming to ensure that the large language model understands the key concepts and entity types involved in landslide events. This part of knowledge includes four parts: entity definition, detailed explanation, distinction from other related entities, and extraction requirements.

[0097] Task description: Describe in detail the role that the large language model needs to play, such as "landslide event extraction expert", and clarify the task goal of the model, that is, to identify and extract specific landslide event entities from text data. At the same time, it also stipulates that the output format is JSON format.

[0098] Landslide event entity description: Use JSON format to describe the landslide event entity, outlining the information contained in the landslide event entity, such as time, location, and attributes.

[0099] Task Example: This aims to demonstrate how to extract landslide event entities from text and how to present the extraction results in a specified format through specific examples. The examples will guide the model to understand the task requirements and provide the correct output format.

[0100] Landslide event text to be extracted: This is the main object of model processing, including news reports, academic papers, government announcements and other texts containing landslide event information.

[0101] 2) Landslide event entity recognition:

[0102] Using the large language model Chat-GLM2-6B, the present invention extracts text information according to the second instruction prompt sentence, and the second instruction prompt sentence input into the large language model includes:

[0103] (1) Knowledge of landslide structure

[0104] "Time of landslide occurrence" refers to the exact time when the landslide accident occurred, accurate to the second.

[0105] "Location of landslide" refers to the specific geographical location where the landslide occurred, as accurate as possible to the street, community or specific landmark.

[0106] "Longitude of the landslide site": refers to the lateral coordinate of the landslide site, for example "116.4074 East longitude", providing accurate longitude coordinates.

[0107] "Causes of landslide formation": the sum of all internal and external factors that lead to landslides. These factors include geological conditions, soil types, hydrological conditions, human activities, etc. It is a comprehensive concept that covers the geological background and cumulative vulnerability, including but not limited to "landslide hazard factors". Note that the causes of landslide formation are more extensive, including not only factors that directly cause landslides, but also a series of indirect and potential inducing factors. It is necessary to extract all internal and external factors that may cause landslides.

[0108] “Primary disasters”: disasters that directly lead to landslides. These disasters are usually direct triggers of landslides, such as earthquakes, rainstorms, etc. Note that they should be distinguished from “concurrent disasters”. Primary disasters focus on the direct causal relationship with landslides, and only the names of disasters are extracted.

[0109] "Secondary disasters": Secondary disasters are secondary disasters caused by landslides. These disasters are the direct consequences of landslides, such as mudslides, barrier lakes, etc. Pay attention to distinguishing them from "primary disasters" and "concurrent disasters". Secondary disasters are chain reactions after landslides. Only the disaster names are extracted.

[0110] (2) Task Description

[0111] Extract events that conform to the schema from the input. Return an empty list if an event does not exist. Return NAN if an argument does not exist. If an argument has multiple values, a list will be returned. Please answer in the format of a JSON string.

[0112] (3) Entity description of landslide event

[0113] "event_type":"Natural Disaster-Landslide";

[0114] "trigger":true;

[0115] "arguments":["Time of landslide","Location of landslide","Longitude of landslide","Latitude of landslide","Number of deaths from landslide","Number of injured from landslide","Number of missing from landslide","Number of trapped from landslide","Number of buried from landslide","Number of affected people from landslide","Landslide hazard factors"………].

[0116] (4) Task examples

[0117] Input: On August 17, 2024, a landslide occurred, causing 5 deaths, 10 missing persons, and direct economic losses of 10 million yuan. The reason was that continuous rainfall caused the soil to be saturated.

[0118] Output:

[0119] {"event_type":"Natural Disaster-Landslide";

[0120] "trigger":true,"

[0121] arguments":{"Time of landslide":"August 17, 2024","Location of landslide":"NAN","Longitude of landslide":"NAN","Latitude of landslide":"NAN","Death toll":"5","Injured toll":"NAN","Missing toll":"10","Trapped toll":"NAN","Buried toll":"NAN","Affected toll":"NAN","Landslide hazard factor":"Continuous rainfall","Cause of landslide":"Continuous rainfall leads to soil saturation"………}.

[0122] (5) Landslide event text to be extracted

[0123] At noon on August 20, 2024, a large-scale landslide occurred. It was initially estimated that more than 30 people were missing and the damage was serious. Experts initially judged that the landslide may be related to the heavy rainfall in recent days.

[0124] Exemplarily, the picture information extraction module includes a picture information preliminary extraction module, a picture extraction result verification module and a picture extraction result correction module;

[0125] The image information preliminary extraction module is used to extract landslide event entities from the landslide event image data based on the third instruction prompt sentence and the large language model to obtain the image data extraction result;

[0126] The image extraction result verification module is used to check whether the image data extraction results extracted from the landslide event image data by the large language model are reasonable and consistent according to the verification instruction prompt statement;

[0127] The image extraction result correction is used to correct erroneous or unreasonable image data extraction results according to the correction instruction prompt statement.

[0128] Specifically, the above describes the specific process of extracting landslide event entities from landslide event text data by the text information extraction module, and here the image information extraction module is introduced.

[0129] Specifically, the image information extraction module has three sub-modules: image information preliminary extraction module, image extraction result verification module and image extraction result correction module. The three sub-modules are introduced below:

[0130] 1) Preliminary image information extraction module:

[0131] In this module, the present invention adopts a detailed extraction strategy: block extraction strategy. The core idea of ​​this strategy is to select a specific set of related entities of the landslide event entity for extraction each time, aiming to avoid the complexity of the instructions and ensure that the model can focus on parsing the image content rather than just understanding the instruction content.

[0132] The specific implementation steps are as follows:

[0133] Entity set definition: The present invention defines multiple entity sets in the feature expression of the aforementioned landslide event entity, such as a semantic description entity set (including landslide event name, landslide event code), a spatial location entity set (including longitude, latitude, detailed address, etc.), and an attribute feature entity set (including landslide grade, landslide type, landslide hazard factor, etc.). The entities in these entity sets are interrelated and reflect the characteristics of landslide events from different angles.

[0134] Picture information extraction instruction prompt: The third instruction prompt sentence of the present invention consists of the following four key parts:

[0135] Structured knowledge: Provide detailed definitions and descriptions of the entities to be extracted to ensure that the large language model understands the specific meaning and extraction criteria of each entity.

[0136] Task description: Clarify the role of the large language model in this task, such as "landslide image information analysis expert", and describe in detail the skills, goals, and output format requirements required to complete the task.

[0137] Extraction hints: Guide the large language model to pay attention to specific visual clues in the image, such as color, shape, texture, etc., to infer the specific value of the entity to be extracted.

[0138] Landslide images: Provide relevant images of the landslide event to be analyzed.

[0139] Large language model application: Using the multimodal large language model GLM-4V-9B, the present invention extracts image information according to the third instruction prompt sentence. An example of the third instruction prompt sentence input into the large language model is as follows:

[0140] (1) Knowledge of landslide structure

[0141] "Landslide length": Landslide length refers to the longest dimension of the landslide body on the horizontal plane, usually measured along the sliding direction of the landslide.

[0142] "Landslide width": The landslide width refers to the width of the landslide body in the horizontal plane, measured perpendicular to the length of the landslide.

[0143] "Landslide thickness": Landslide thickness refers to the vertical thickness of the landslide body, that is, the vertical distance between the landslide body from the ground to the sliding surface.

[0144] "Landslide height": Landslide height usually refers to the relative height of the landslide body, that is, the vertical distance from the sliding surface to the ground.

[0145] "Landslide area": ​​Landslide area refers to the projection area of ​​the landslide body on the ground, that is, the surface area covered by the landslide body.

[0146] (2) Task Description

[0147] You are an expert who focuses on extracting relevant landslide event information from landslide images. Please analyze the image content and only report the features that actually exist in the image. The output format is a JSON object in the format of {output_format}.

[0148]

[0149] (3) Extraction Tips

[0150] Use the ruler in the image or the known size of objects (e.g. houses, trees, vehicles, people, etc.) as a reference to estimate the size of the relevant attributes of the landslide in meters. If the image is not possible to estimate, please return NAN and specify the specific size of the reference object.

[0151] (4) Pictures

[0152] . / img / img_1.jpg

[0153] 2) Image extraction result verification module:

[0154] In order to ensure that the extraction result is consistent with the image content and the reasoning process is logical, the present invention designs a set of verification instruction prompt statements to check whether the information extracted from the image by the image information preliminary extraction module is reasonable and consistent.

[0155] The main components of the verification instruction prompt statement are as follows:

[0156] Task description: In this section, the invention details the role of the large language model in the verification task, such as "image information logic verification expert". At the same time, the skills, goals and output formats required to complete the task are defined.

[0157] Image extraction results to be verified: This part includes the information that the model has previously extracted from the image, which will be used as input for verification. For example, if the model extracts the length of the landslide, then this length value is the result to be verified.

[0158] The following is an example of a validation command prompt statement for the input model:

[0159] (1) Task Description

[0160] You are an expert who focuses on judging whether the image extraction results are correct. Your task is to judge whether the results extracted from the image are correct. Note: If the 'reasoning process' reasonably explains why no specific value can be obtained from the image, then even if the 'extraction result' is 'NAN', it should be considered consistent.

[0161] Output format (JSON):

[0162]

[0163] (2) Image extraction results to be verified

[0164]

[0165] 3) Image extraction result correction module:

[0166] During the verification process of the image information extraction results, if the results are found to be illogical or erroneous, they need to be corrected. The present invention designs a set of correction instruction prompts to guide the image extraction result correction module to correct erroneous or unreasonable extraction results. These prompts ensure that the model can accurately identify and correct errors.

[0167] The main components of the modified instruction prompt statement are as follows:

[0168] Task description: In this section, the invention details the role of the large language model in the correction task, such as "image information correction expert". At the same time, the skills, goals and output formats required to complete the task are clearly stated.

[0169] Image extraction results to be corrected: This part includes the information that the model previously extracted but failed to pass the verification, and also includes the reasons for failure to pass the verification obtained from the "Image Extraction Verification Module".

[0170] Images: Provide links to original images or embed images so that the model can directly reference the image content during the correction process.

[0171] An example of a correction instruction prompt statement for the input model is as follows:

[0172] (1) <Task Description>

[0173] You are an expert who focuses on correcting the content of image information extraction. Your task is to correct the 'extraction process' and 'reasoning process' according to the error causes of the 'extraction results' and 'reasoning process'. The output format is a JSON object in the {output_format} format.

[0174]

[0175] (2) Image extraction results to be corrected

[0176]

[0177]

[0178] (3) Pictures

[0179] . / img / img_1.jpg

[0180] Step 4: fuse the text data extraction result and the image data extraction result to obtain a landslide event entity extraction fusion result.

[0181] The above-mentioned text information extraction module and image information extraction module are used to extract landslide event entities from landslide event text data and landslide event image data respectively, and obtain text data extraction results and image data extraction results respectively. For the text data extraction results and image data extraction results, the two results need to be fused.

[0182] Exemplarily, the result fusion module includes a digital result fusion module and a text description result fusion module, and the text data extraction result and the image data extraction result are fused based on the result fusion module, including:

[0183] For the digital extraction results, based on the digital result fusion module, determine whether the text data extraction results and the image data extraction results are consistent. If they are consistent, add the image data supporting annotation to the text data extraction results. If they are inconsistent, the text data extraction results shall prevail, wherein the digital extraction results refer to the results that can be directly quantified;

[0184] For the description extraction result, according to the fusion instruction prompt statement, based on the text description result fusion module, the text data extraction result and the image data extraction result are fused to generate a fused description result, wherein the description extraction result refers to a result with descriptive information;

[0185] The fusion instruction prompt statement includes a task description, a text data extraction result to be fused, and a picture data extraction result to be fused.

[0186] Specifically, when fusing text and image extraction results, the present invention adopts a classification fusion strategy to ensure the accuracy and consistency of the results.

[0187] 1) Digital result fusion module:

[0188] For results that can be directly quantified, such as "number of deaths from landslides", "number of people affected by landslides", and "amount of landslide losses", the text data extraction results are preferred because text data is often more accurate. If the image data extraction results are the same as the text data extraction results, the "picture evidence" annotation is added to the final result to enhance the credibility of the information. If the image data extraction results are different from the text data extraction results, the text data extraction results shall prevail.

[0189] 2) Text description result fusion module:

[0190] For results with descriptive information, such as "rescue activities", "warning situation" and "resident losses", the present invention designs a set of "fusion instruction prompts" to integrate the text data extraction results and the image data extraction results to generate a more comprehensive and accurate description. The main components of the fusion instruction prompt are as follows:

[0191] Task description: Specify the role of the model in the fusion task, such as "information fusion expert". Detail the goal of the fusion task, which is to combine text and image information to generate a comprehensive description. Specify the format and structure of the output results.

[0192] Text extraction results to be fused: Provide the results extracted from text data.

[0193] Image extraction results to be fused: Provides the results extracted from the image data.

[0194] An example of a fusion instruction prompt statement for the input model is as follows:

[0195] (1) Task Description>

[0196] You are an expert in information fusion. Your task is to combine the following text and image extraction results to generate a comprehensive and accurate descriptive information. Please ensure that the output description is clear, coherent, and reflects the actual situation of the landslide event.

[0197] Output format (JSON):

[0198]

[0199] (2) Text extraction results to be fused

[0200] "Rescue activities":"Search and rescue, medical treatment"

[0201] "Housing damage": "The restaurant of a guesthouse was washed away"

[0202] (3) Image extraction results to be fused

[0203] {"rescue activity":"search and rescue dog searching","reasoning":"There are several people in orange uniforms and one in blue uniform in the picture, who may be rescuers. There is a search and rescue dog searching in the middle left of the picture, so it is inferred that a search and rescue activity is underway."}

[0204] {House damage: "Partial damage to the house", "Reasoning": "The pictures show that the house has collapsed walls and a missing roof, so it is judged to be partially damaged."}

[0205] A multimodal landslide event entity extraction method provided by an embodiment of the present invention first constructs a landslide entity expression model with "feature expression and relationship expression" as the core, and then proposes a command prompt-driven cross-platform multimodal landslide event data set construction method. The method develops special command prompts and combines online annotation to construct a multimodal landslide event data set. Then, a multimodal landslide event entity extraction model based on prompt engineering is constructed. The model is based on the prompt engineering concept and achieves an in-depth understanding of landslide events through a set of command prompt statements that integrate structured knowledge. Finally, a comprehensive description of landslide event entities is performed through a classification fusion strategy. The present invention effectively solves the problems of low efficiency in existing landslide event data processing, low accuracy in entity extraction, and difficulty in multimodal data fusion, and provides strong support for landslide disaster warning and emergency management.

[0206] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0207] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0208] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-modal landslide event entity extraction method, characterized in that: include: Step 1, establishing a landslide event entity and entity relationship expression framework to establish a landslide entity expression model, wherein the landslide event entity and entity relationship expression framework includes a feature expression system of the landslide event entity and a relationship expression system of the landslide time entity; Step 2, based on the established landslide entity expression model, using automatic crawler technology to obtain multimodal landslide event data from multiple channels, and preprocessing and annotating the multimodal landslide event data based on instruction prompt driving, wherein the multimodal landslide event data includes landslide event text data and landslide event image data; Step 3, constructing a multimodal landslide event entity extraction model based on prompt engineering, extracting the landslide event text data and the landslide event image data respectively, and obtaining a text data extraction result and an image data extraction result; Step 4: fuse the text data extraction result and the image data extraction result to obtain a landslide event entity extraction fusion result.

2. The multi-modal landslide event entity extraction method according to claim 1 is characterized in that: The establishment of a landslide event entity and entity relationship expression framework to establish a landslide entity expression model includes: Based on the semantic description, spatial location, temporal evolution and attribute characteristics of landslide event entities, a feature expression system of landslide event entities is established, including: The formal expression of the semantic description of the landslide event entity is as follows: AND semantics ={And name ,AND ID } Among them, E semantics represents the landslide semantic description entity, E name Indicates the name of the landslide event, E ID Indicates the unique identification code of the landslide event; The formal expression of the spatial location of the landslide event entity is as follows: AND space ={And longitude ,AND latitude ,AND admiArea ,AND address ,AND admiID } Among them, E space represents the spatial location entity of the landslide event, E longitude Indicates the longitude of the landslide event; E latitude Indicates the latitude of the landslide event; E admiArea The administrative area of ​​the landslide event mainly refers to: the smallest external administrative area that can completely include the spatial location of the landslide event; E address Indicates the detailed address of the landslide event to the village or community; E admiID Indicates the administrative region code, which is a standardized expression of administrative division areas; The time evolution of a landslide event can be formally expressed as follows: AND Time ={And OCtime ,AND evolution1 ,AND evolution2 ,AND evolution3 …} Among them, E Time represents the time evolution entity of the landslide event, E OCtime Indicates the time of occurrence of landslide event; E evolution1 Indicates the first evolution time of the landslide event; E evolution2 Indicates the second evolution time of the landslide event; E evolution3 represents the third evolution time of the landslide event; The formal expression of the attribute characteristics of landslide events is as follows: Among them, E level Indicates the landslide grade; E type Indicates the landslide type; E factors Indicates landslide hazard factor; E environment Indicates landslide disaster-prone environment; E suffer Indicates the landslide hazard-bearing body; E warning Indicates pre-disaster warning; E rescue Indicates disaster relief; E reconstruction Indicates post-disaster reconstruction; E geometry Represents the geometric characteristics of the landslide; E dynamic Indicates the dynamic characteristics of landslide; Based on the hazard relationship, spatial correlation relationship, temporal correlation relationship and attribute correlation relationship of landslide events, a relational expression system of landslide time entities is established, including: The hazard relationship of landslide events is defined as: R disaster ={R PriDisaster ,R ConcDisaster ,R SeconDisaster } Where R PriDisaster represents primary disaster, R ConcDisaster represents concurrent disasters, R SeconDisaster Indicates secondary disasters; The spatial correlation relationship of landslide events is defined as: R space ={R topology ,R distance ,R direction } Among them, R space is the spatial relationship of landslide, R topology is the spatial topological relationship; R direction Indicates the orientation relationship, R direction Represents the distance between landslide event entities; The temporal correlation of landslide events is defined as: R time ={R year ,R month } Among them, R time is the landslide time relationship; R year R is the time sequence of the year; month Indicates the time sequence of months; The attribute association relationship of landslide events is defined as: Among them, R level Represents the similarity relationship of landslide grade; R type Represents the similarity relationship of landslide types; R factors Represents the similarity relationship between landslide hazard factors; R environment Represents the similarity relationship of landslide disaster-prone environment; R suffer Represents the similarity relationship of landslide hazard-bearing bodies; R warning Represents the similarity relationship of pre-disaster warning; R rescue Indicates disaster relief; R reconstruction Represents the similarity relationship of post-disaster reconstruction; R geometry Represents the similarity relationship of landslide geometric characteristics; R dynamic Represents the similarity relationship of landslide dynamic characteristics.

3. The multi-modal landslide event entity extraction method according to claim 1 is characterized in that: The preprocessing and labeling of the multi-modal landslide event data based on the instruction prompt drive includes: Preprocessing the landslide event text data, wherein the preprocessing includes removing irrelevant information, segmenting words, and removing stop words; For the fuzzy time words and fuzzy space words in the landslide event text data, a first instruction prompt sentence is set, wherein the first instruction prompt sentence instructs the large language model to understand the task objectives, processing methods and output formats, and the first instruction prompt sentence includes a task description, a conversion example and text data to be processed, and guides the large language model to convert the fuzzy time words and fuzzy space words in the landslide event text data based on context information; Performing image noise reduction, format conversion and size adjustment on the landslide event image data; Entity types and annotation rules are defined, and the processed landslide event text data and landslide event image data are annotated based on the defined entity types and annotation rules.

4. The multi-modal landslide event entity extraction method according to claim 3 is characterized in that: The multimodal landslide event entity extraction model includes a text information extraction module, a picture information extraction module and a result fusion module. The step 3 constructs a multimodal landslide event entity extraction model based on the prompt engineering, extracts the landslide event text data and the landslide event picture data respectively, and obtains text data extraction results and picture data extraction results, including: A second instruction prompt sentence for extracting landslide event entities is constructed based on the text information extraction model, and according to the second instruction prompt sentence, landslide event entities are extracted from landslide event text data based on a large language model to obtain a text data extraction result; Constructing a third instruction prompt sentence for extracting landslide event entities based on the picture information extraction model, and extracting landslide event entities from landslide event picture data based on the large language model according to the third instruction prompt sentence to obtain picture data extraction results; The text data extraction result and the image data extraction result are fused based on the result fusion module.

5. The multi-modal landslide event entity extraction method according to claim 4 is characterized in that: The second instruction prompt sentence includes landslide structured knowledge, task description, landslide event entity description, task example and landslide event text data to be extracted, wherein the landslide structured knowledge includes entity definition, detailed explanation, distinction from other related entities and extraction requirements, the task description includes the role that the large language model needs to play, task objectives and output result format, the landslide event entity description includes the time, location and attributes of landslide event mapping, and the task example includes a method of extracting landslide event entities from landslide event text data and a method of presenting extraction results through specific examples; The method extracts landslide event entities from landslide event text data based on the large language model to obtain text data extraction results, including: According to the second instruction prompt sentence, based on the large language model Chat-GLM2-6B, the landslide event entity is extracted from the landslide event text data to obtain the text data extraction result.

6. The multi-modal landslide event entity extraction method according to claim 4 is characterized in that: The picture information extraction module includes a picture information preliminary extraction module, a picture extraction result verification module and a picture extraction result correction module; The image information preliminary extraction module is used to extract landslide event entities from the landslide event image data based on the third instruction prompt sentence and the large language model to obtain the image data extraction result; The image extraction result verification module is used to check whether the image data extraction results extracted from the landslide event image data by the large language model are reasonable and consistent according to the verification instruction prompt statement; The image extraction result correction is used to correct erroneous or unreasonable image data extraction results according to the correction instruction prompt statement.

7. The multi-modal landslide event entity extraction method according to claim 6 is characterized in that: The third instruction prompt sentence includes structured knowledge, task description, extraction prompt sentence and to-be-extracted, wherein the extraction prompt sentence instructs the large language model to focus on specific visual clues in the image data to infer the specific value of the entity to be extracted; Based on the large language model, landslide event entities are extracted from the landslide event image data to obtain the image data extraction results, including: According to the third instruction prompt statement, based on the multimodal large language model GLM-4V-9B, the landslide event entity is extracted from the landslide event image data to obtain the image data extraction result.

8. The multi-modal landslide event entity extraction method according to claim 6 is characterized in that: The verification instruction prompt statement includes a task description and a picture data extraction result to be verified, and the correction instruction prompt statement includes a task description and a picture data extraction result to be corrected.

9. The multi-modal landslide event entity extraction method according to claim 4, characterized in that: The result fusion module includes a digital result fusion module and a text description result fusion module, and the text data extraction result and the image data extraction result are fused based on the result fusion module, including: For the digital extraction results, based on the digital result fusion module, determine whether the text data extraction results and the image data extraction results are consistent. If they are consistent, add the image data supporting annotation to the text data extraction results. If they are inconsistent, the text data extraction results shall prevail, wherein the digital extraction results refer to the results that can be directly quantified; For the description extraction result, according to the fusion instruction prompt statement, based on the text description result fusion module, the text data extraction result and the image data extraction result are fused to generate a fused description result, wherein the description extraction result refers to a result with descriptive information; The fusion instruction prompt statement includes a task description, a text data extraction result to be fused, and a picture data extraction result to be fused.

Images