Dam monitoring analysis method and system based on multi-modal knowledge graph and large language model

Abstract

The invention provides a dam safety monitoring data analysis method and system based on a multi-modal knowledge graph and a large language model, and belongs to the technical field of calculation models. The method comprises the following steps: collecting multi-modal data of dam monitoring, and preprocessing the multi-modal data; constructing a dam safety analysis knowledge graph by adopting a water conservancy object multi-modal knowledge graph construction mode; an intelligent positioning function is used for finding an associated object for the monitored abnormal signals, information of the associated object is obtained, and a dam analysis model is used for association analysis; generating an abnormal associated sub-graph; and forming a multi-dimensional analysis report conclusion by using a pre-trained water conservancy professional knowledge large language model in combination with cue words and user questions. Multi-aspect information generated in the dam safety management process is integrated by utilizing the knowledge graph to form a comprehensive information network, efficient query and reasoning are realized by virtue of a graph computing technology, and a monitoring analysis result for explaining the dam is clearly, intuitively, accurately and comprehensively presented.

Description

Technical Field

[0001] This invention belongs to the field of computational models and relates to artificial intelligence technology, specifically to a method and system for dam monitoring and analysis based on multimodal knowledge graphs and large language models. Background Technology

[0002] Traditional dam safety monitoring and analysis mainly relies on single-modal data obtained from sensor monitoring. After the monitoring data is analyzed and evaluated by professional technicians using methods such as comparison, plotting, eigenvalue statistics and mathematical modeling, the operational status of the dam monitoring points is obtained. Combined with information from manual inspections, the overall safety of the dam is assessed. This approach has a high degree of subjectivity or dependence on experts.

[0003] With the development of information technology, an increasing number of dam safety information management systems are being applied to the daily safety supervision of large and medium-sized reservoirs. These systems combine statistical analysis, numerical simulation, and machine learning methods to conduct quantitative analysis of dam safety, but manual intervention and experience-based judgment are still required. In addition, some systems transform expert experience and knowledge into rules, building expert systems to assist in safety assessments and even incorporating them into intelligent safety analysis and early warning applications within digital twin projects. However, these analytical methods struggle to fully utilize the complementarity of multimodal data (such as images, videos, and text logs) within the system, and also fail to form a comprehensive and systematic analytical perspective. This results in biased and inaccurate analysis results. Furthermore, data analysis results are typically presented in chart form, lacking scientific interpretation and decision-making recommendations, leading to a poor user experience. Summary of the Invention

[0004] To address the problems of insufficient utilization of multimodal data, shallow modeling of correlation relationships, and poor readability of analysis results in existing technologies, this invention proposes a method and system for analyzing dam safety monitoring data based on multimodal knowledge graphs and large language models, enabling in-depth correlation analysis and natural language interpretation of multimodal data.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] A method for analyzing dam safety monitoring data based on multimodal knowledge graphs and large language models includes the following steps:

[0007] Step 1: Collect multimodal data for dam monitoring and preprocess the multimodal data;

[0008] Step 2: Construct a dam safety analysis knowledge graph using a multimodal knowledge graph construction method for water conservancy objects;

[0009] Step 3: Use the intelligent agent to extract multimodal features, update the knowledge graph based on incremental data, use the intelligent positioning function to find the associated objects of the detected abnormal signals and obtain the information of the associated objects, and apply the dam analysis model to perform association analysis.

[0010] Step 4: Based on the correlation analysis results obtained in Step 3, generate an abnormal correlation subgraph, which includes: abnormal points and correlation points;

[0011] Step 5: Transform the abnormal correlation subgraph generated in Step 4 into a graph description language for input into the large language model. Use multimodal data information to supplement features and form structured large model prompt words. Use a pre-trained large language model with water conservancy expertise, combined with prompt words and user questions, to form a multidimensional analysis report conclusion.

[0012] Furthermore, in step 1, the multimodal data includes: sensor monitoring data, images, videos, and text logs, and the preprocessing includes: cleaning, denoising, and standardizing the data.

[0013] Furthermore, in step 3, the specific steps for achieving intelligent positioning are as follows:

[0014] First, the various ontologies involved in the water conservancy object and the relationships between them are constructed. Then, location information is added to the attributes of each ontology to build a knowledge graph data. Next, when a query is input, the entity name and the distance range to be queried are extracted from the query, and the distance between the entity and other water conservancy objects is calculated. Finally, entities within the range that meet the conditions are filtered out, and the connection relationship between the target entity and these entities is obtained. Finally, the entities and the connection relationship between entities are presented on the interface.

[0015] Furthermore, in step 5, the multimodal data information includes: sensor data features obtained through API interfaces, and associated with image / text summaries.

[0016] Furthermore, in step 5, converting the anomaly association subgraph generated in step 4 into a graph description language for input to the large language model specifically includes:

[0017] This paper summarizes the common and important features of the abnormal association combination analysis graph, transforms these features into natural language, summarizes the general transformation template f(x) from the natural language, uses the transformation module to map subsequent feature data into natural language, and combines the transformed natural language with corresponding prompts and knowledge to enable the large language model or knowledge platform to generate the corresponding graph description language.

[0018] Furthermore, it also includes:

[0019] Step 6: Present the combined analysis chart and multidimensional analysis report conclusions in a visual format, and provide interactive functions, allowing users to dynamically adjust the combination of analysis charts, view detailed information and export reports. It also supports calling the knowledge platform query robot to refer to similar historical cases.

[0020] The present invention also provides a dam safety monitoring data analysis system based on multimodal knowledge graphs and large language models, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the dam safety monitoring data analysis method based on multimodal knowledge graphs and large language models.

[0021] The beneficial effects of this invention are as follows:

[0022] 1. This invention integrates the advantages of traditional knowledge graph visualization and interpretability with the advantages of large language models in terms of strong unstructured information processing capabilities and domain generalization. It utilizes knowledge graphs to integrate various information generated during dam safety management, forming a comprehensive information network. With the help of graph computing technology, it achieves efficient querying and reasoning, promotes the fusion of multi-source heterogeneous data, knowledge computing, knowledge reasoning, and knowledge visualization capabilities, and can clearly, intuitively, accurately, and comprehensively present and interpret the monitoring and analysis results of dams. It makes up for the pain points of the original dam monitoring and analysis being difficult, difficult to apply, and dependent on experts, and realizes intelligent analysis and early warning of dams.

[0023] 2. The solution of this invention can realize cross-business data collaboration and joint reasoning, promote the upgrade of reservoir safety management from "single point response" to "full-domain linkage", facilitate further promotion, and provide innovative cases for reservoir safety management.

[0024] 3. This invention utilizes a large language model and intelligent agent for technological innovation, innovating the joint computational reasoning technology of graph networks and parametric semantic networks. This enables the system to possess interpretable and controllable GOT (Graph Thinking) capabilities, breaking the limitations of static data on model cognition, and constructing a multimodal graph capability of "perception-cognition-decision," thereby realizing intelligent analysis for dam safety monitoring. Attached Figure Description

[0025] Figure 1 The flowchart of the dam monitoring and analysis method based on multimodal knowledge graph and large language model provided by this invention.

[0026] Figure 2 The system architecture diagram for implementing the solution of this invention.

[0027] Figure 3 Construct a data architecture diagram for a multimodal water conservancy object knowledge graph.

[0028] Figure 4 This is a diagram of the architecture of a large-scale intelligent agent.

[0029] Figure 5 This is a diagram of the intelligent positioning interface.

[0030] Figure 6 This is a functional structure diagram of the knowledge engine.

[0031] Figure 7 The process of implementing water engineering management analysis methods.

[0032] Figure 8 This is an abnormal correlation sub-graph, which includes a field diagram. Detailed Implementation

[0033] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only for illustrating the present invention and are not intended to limit the scope of the present invention. In addition, the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] The architecture of this invention is as follows Figure 2 As shown, this method integrates multiple platforms (data platform, knowledge platform, and model platform) and can be applied to water conservancy and hydrological management scenarios such as engineering management, watershed management, and reservoir management. The dam safety monitoring data analysis method based on multimodal knowledge graphs and large language models provided by this invention has the following process: Figure 1 As shown, it includes the following steps:

[0035] Step 1: Multimodal Data Acquisition and Preprocessing

[0036] Multimodal data from dam monitoring was collected and preprocessed, including cleaning, noise reduction, and standardization. The multimodal data included sensor monitoring data, images, videos, and text logs.

[0037] Step 2: Construction of Multimodal Knowledge Graph

[0038] The dam safety analysis knowledge graph is constructed using a multimodal knowledge graph construction method for water conservancy objects, and its data architecture is as follows: Figure 3 As shown, the construction process includes the following steps:

[0039] (1) Construct an entity classification system

[0040] Following professional guidance from water conservancy experts and in accordance with industry standards, a classification system was constructed and classification labels were established. In this example, the industry standard adopted is the "General Rules for Classification and Coding of Water Conservancy Information" (SL / T 701-2021), and the classification is directly based on the "General Rules for Classification and Coding of Water Conservancy Information" to form a classification system for dam safety analysis entities.

[0041] (2) Identification of basic and business ontology in multimodal knowledge graph

[0042] The basic ontology refers to the ontology of water conservancy and hydraulic engineering objects, including common types of reservoirs, dams, rivers and lakes in the water conservancy field; the business ontology refers to the ontology built on the basis of related business, such as the analysis methods (analysis method ontology) and analysis indicators (analysis indicator ontology) involved in the safety analysis of dams in dam safety analysis.

[0043] (3) Declarative and procedural knowledge representation

[0044] Declarative and procedural knowledge refers to the relationships between ontologies. For example, in safety analysis, the relationship between the dam ontology and the indicator ontology could be "safety analysis indicators include", so that an edge of the type "dam ontology - [safety analysis indicators include] -> indicator ontology" can be formed in the knowledge graph.

[0045] (4) Description of activities and patterns in the water conservancy field

[0046] Activities and patterns in the water conservancy field can be reflected in relationships, such as "a certain river - [inflow] -> dam".

[0047] Steps (3) and (4) establish the relationships between ontologies, including the relationships between basic ontologies, between basic ontologies and business ontologies, and between business ontologies.

[0048] (5) Construction of knowledge graph of water conservancy objects.

[0049] Structured knowledge is constructed by extracting entity-relationship-attribute triples from the multimodal data collected and processed in step one, thus building a knowledge graph of water conservancy objects. The resulting multimodal knowledge graph is: an entity set. Inclusion relation triples ,in, For entities, In the dam ontology, an entity refers to a specific instance of the dam ontology, such as "Chaohe Main Dam"; a relation refers to a connection between other instances, such as the "inclusion" relationship between "Miyun Reservoir" and "Chaohe Main Dam" in the reservoir ontology; an attribute refers to the attribute information existing in the ontology or relation (e.g., the dam ontology contains attributes such as "dam length" and "dam height"). For unstructured knowledge, knowledge objects are sliced, and entities and triples are extracted from the slices. The result is: an entity set. Attached attributes (such as text descriptions and type tags) are included. Knowledge object slices and multimodal knowledge graphs are aligned and integrated through entities and relationships between entities to establish unified associations between data, achieving timeliness, spatiotemporal alignment, and semantic alignment of multimodal data. The constructed knowledge graph is used to associate with the dam safety analysis knowledge base.

[0050] The main steps of alignment and blending are:

[0051] Step 1: Alias ​​matching based on regular expression rules

[0052] (1) Predefine a list of entity aliases within the domain and construct a domain-related alias mapping table. In this context, term represents an entity, alias represents an alias, and p represents the number of entities. For example, Miyun Reservoir is aliased as Miyun. When performing alignment and merging later, Miyun Reservoir and Miyun are essentially the same entity.

[0053] (2) Matching text variations of entities in different modalities (such as abbreviations, synonyms, etc.) using regular expressions, specifically including:

[0054] a. to and The entity text descriptions in the document are normalized (e.g., lowercase, punctuation removed), where, For an entity of mode s, For an entity of mode k;

[0055] b. Using regular expressions Match aliases, if If true, then it is marked as a potential alignment pair. ,in, Let i be the entity, and Regex be the regular expression. This refers to performing regular expression filtering and matching on entity i, where Match() is the result of the regular expression matching algorithm. Represents entity text strings, Represents another entity text string, This represents a pair of two entities.

[0056] Step 2: Similarity calculation based on embedding model

[0057] (1) Use pre-trained embedding models (such as BERT, TransE, etc.) to embed two different entities and Encoded as vector V e V e’ For multimodal entities, visual feature vectors are extracted and mapped to the same semantic space.

[0058] (2) Semantic similarity between entities is measured using cosine similarity or Euclidean distance, and entity pairs with similarity exceeding a preset threshold are aligned. For example, using cosine similarity:

[0059] The formula for calculating cosine similarity is as follows:

[0060]

[0061] like If a pair is selected based on a preset threshold (e.g., 0.85), it is marked as a candidate alignment pair.

[0062] Step 3: Alignment and Decision Fusion

[0063] (1) Comprehensive scoring: Combining the rules and model results, the final alignment score is calculated using the following weighted formula:

[0064]

[0065] in The result is the rule matching result (0 / 1 binary value), and λ is the weight (e.g., 0.3).

[0066] During the fusion process, higher-order relationships in the multimodal graph are retained first to enhance data consistency.

[0067] (2) Conflict resolution: If there are multiple candidate alignments for the same entity, the one that is selected first shall be chosen first. The highest pairing that exceeds the threshold.

[0068] Finally, we obtain the aligned entity set. The merged multimodal knowledge graph G' adds cross-modal relation edges. .

[0069] Step 3: Multimodal Feature Extraction and Correlation Analysis

[0070] Applying large-scale intelligent agents (architecture such as) Figure 4 (As shown) Technological innovation is carried out by dividing the roles of intelligent agents, including establishing four intelligent agent roles: memory, tool, action, and planning. These agents are used to achieve multimodal feature extraction and correlation analysis in business applications. The memory agent is responsible for knowledge retrieval and state maintenance, enabling rapid retrieval of subgraph indexes and progressive construction of the knowledge graph; the tool agent encapsulates water conservancy professional functions; the action agent is responsible for specific task execution; and the planning agent coordinates the entire decision-making process, forming a closed-loop management of thinking, reasoning, and feedback.

[0071] The division of labor and collaboration among intelligent agents is an important research direction in the field of multi-agent systems in artificial intelligence. Specifically, it involves a method for implementing role division based on a workflow and agent collaborative architecture, suitable for automating complex tasks. A typical agent architecture usually includes four core modules: planning and thinking, decision selection, action execution, and observation and perception. The planning and thinking module is responsible for decomposing complex tasks and generating sub-goals, providing structured guidance for subsequent execution by constructing multi-stage action paths. The decision selection module, based on the paths generated in the previous stage, makes tool selection decisions. The action execution module is responsible for calling the selected tools and outputting the execution results. The observation and perception module analyzes the tool execution results, forming a final conclusion or triggering a cyclical iterative mechanism of "planning and thinking → decision selection → action execution → observation and perception."

[0072] At the business application level, multimodal feature extraction and association analysis functions can be achieved by deploying intelligent agents. Specifically, firstly, a multimodal feature extraction intelligent agent is constructed, which completes the extraction of professional entities and the modeling of association relationships based on a pre-defined water conservancy professional ontology; then, an association analysis intelligent agent is deployed, which performs link association analysis between entities according to preset entity types and relationship types, thereby supporting the intelligent processing of complex business scenarios in the water conservancy field.

[0073] A general framework for agent collaboration includes

[0074] 1. Workflow Cycle Formula

[0075] System state(t) = Planning (task) → Decision (tool) → Execution (input) → Observation (result)

[0076] In this invention, the symbol mapping rule is as follows:

[0077] • Task (θ): For example, "Risk Assessment of Water Conservancy Projects" can be broken down into {Data Acquisition, Model Calculation, Report Generation}.

[0078] • Tools (φ): Decision function: Select "Hydrological Simulator" or "GIS Analysis Tool" from the tool library.

[0079] • Input (ψ): Parameters required by the tool, such as rainfall data, topographic maps, etc.

[0080] • Result (Ω): Structured data of execution output / error log

[0081] The intelligent agent analyzes unstructured documents (pdf, docx, text, etc.) uploaded by users and transforms this unstructured content into structured knowledge (triplets, text fragments). This knowledge is used as key information in intelligent positioning and intelligent question answering.

[0082] 2. The dynamic update rules are as follows:

[0083]

[0084] Applying intelligent agents to extract multimodal features includes:

[0085] 1. Feature Generation Formula

[0086] Feature data = extracted entities (obtained from an established ontology library) + modeled relationships (obtained from existing multimodal data)

[0087] Symbol instantiation

[0088] • Ontology (O): Terminology tree for the water conservancy field (e.g., "reservoir" → "dam" → "spillway")

[0089] • Multimodal data (M):

[0090] • Text: "Water level exceeds warning line by 0.5 meters" in the flood control report.

[0091] • Image: Water body boundary in satellite remote sensing

[0092] • Sensor: Real-time data stream from the flow meter

[0093] • Entity set (E): {"Three Gorges Dam", "Yangtze River", "2024 Flood Season"}

[0094] • Relational triple (R): ["Three Gorges Dam - located on the Yangtze River", "Yangtze River - water level - 2024 flood season"]

[0095] 2. Knowledge graph update

[0096] When new knowledge data is uploaded to the system, the knowledge graph is updated based on the knowledge graph obtained in step one:

[0097] New knowledge = Retrieving old knowledge (query) + Merging new discoveries (incremental data)

[0098] Operation Example

[0099] • Search: Enter the query "water conservancy facilities around Dongting Lake" → Return to the sub-map {Dongting Lake dikes, Yueyang sluice gate}

[0100] • Merge: Add a new relationship "Yueyang Sluice Gate - Regulation - Dongting Lake Water Level"

[0101] The association analysis phase includes:

[0102] 1. Correlation strength calculation

[0103] Association Score = ∑ (Similarity between Entity A and Entity B | Relationship Type Constraint)

[0104] • Business Scenario Examples

[0105] • Relation type T_r: "Geographical affiliation"

[0106] ·calculate:

[0107] • Similarity ("Dongting Lake", "Hunan Province" | "Affiliation") = 0.9

[0108] • Similarity ("Dongting Lake", "Yangtze River" | "Water System Connectivity") = 0.8

[0109] Output: Prioritize establishing the association "Dongting Lake - belongs to - Hunan Province"

[0110] Targeting spatiotemporal characteristics, this method combines an expert experience base, an engineering safety knowledge base, and a pre-constructed dam safety analysis knowledge graph. Utilizing intelligent positioning, it identifies associated objects (objects with association scores exceeding a set threshold) based on monitored abnormal signals (such as excessive displacement). It then obtains profile information and data retrieval interfaces for these associated objects and applies a dam analysis model for association analysis and reasoning. The dam analysis model is based on existing model analysis algorithms from digital twin hydraulic engineering model libraries.

[0111] The specific steps for achieving intelligent positioning are as follows:

[0112] First, we construct the various entities and relationships (formulas and descriptions, such as triples like (tunnel, reservoir, hydrological station, spillway, etc.) involved in water conservancy objects (e.g., tunnel, located at, dam), (reservoir, contains, tunnel), (reservoir, inflow, river), etc.) involved in the water conservancy objects. Then, we add location information to the attributes of each entity. In this invention, the location information is their latitude and longitude information, and we construct a knowledge graph data. Then, when a query is input, we extract the entity name and the distance range (km) to be queried from the query (e.g., if the user's question is: What hydraulic engineering facilities are there within 50 kilometers of Miyun Reservoir? Then, we need to extract the location entity: Miyun Reservoir, which is used for subsequent retrieval in the knowledge graph). We calculate the distance between the entity and other water conservancy objects based on latitude and longitude. Finally, we filter out the entities within the range that meet the conditions and obtain the connection relationship between the target entity and these entities. Finally, we present this intuitively on the interface, such as... Figure 5 As shown.

[0113] Step 4: Dynamically generate combined analysis diagrams

[0114] like Figure 6As shown, based on a data platform (providing water conservancy data support, such as sensor data obtained through API interfaces), a knowledge base, and business events (user input), a dynamic generation of a combined analysis graph is achieved through a dual-drive approach of "knowledge graph - large-scale intelligent agent," encompassing knowledge query, question answering, calculation, and generation. The overall process is as follows: Figure 7 As shown.

[0115] Taking dam safety monitoring and analysis as an example: Based on the entity relationships of abnormal locations within the dam, correlation analysis is performed. Various information from highly relevant points (related objects) obtained based on knowledge graphs are combined and displayed, dynamically generating anomaly correlation sub-graphs (containing combinations of various analysis graphs of abnormal points and related points). This provides a more intuitive graphical analysis basis for the rapid analysis and judgment of dam monitoring anomalies. Anomaly correlation sub-graphs are shown below. Figure 8 As shown in the image, the picture shows the footage captured by camera #13 at the associated location.

[0116] Step 5: Large Language Model-Assisted Decision Analysis

[0117] (1) Transform the anomaly correlation subgraph generated in step four into an input format that the large language model can understand (including anomaly locations, propagation paths, and multimodal data features), and use a graph description language to express entity relationships, for example:

[0118] [Dam Section B3] - [Pierre gauge UP17 reading exceeds threshold] > [Upstream water level SW, piezometer readings at the same and nearby sections show no abnormal changes]. Here, [Dam Section B3] is the anomalous point, and the data for [Pierre gauge UP17 reading exceeds threshold] and [Upstream water level SW, piezometer readings at the same and nearby sections show no abnormal changes] are multimodal data features of the corresponding propagation path and highly correlated points. The ">" sign indicates the propagation path direction is from left to right.

[0119] [Pyrometer UP17] - [Query for Anomalies at Other Types of Measuring Points within 100 Meters] > [Weir WE1: 0.26 L / s]. Where [Pyrometer UP17] is the anomaly point, [Query for Anomalies at Other Types of Measuring Points within 100 Meters] and [Weir WE1: 0.26 L / s] are the multimodal data characteristics of the corresponding propagation path and highly correlated points, and ">" indicates the propagation path direction is from left to right.

[0120] [Pyrometer UP17] - [Correlation Coefficient: 0.85] > [Weir WE1: 0.26 L / s]. Here, [Pyrometer UP17] is an outlier, [Correlation Coefficient: 0.85] and [Weir WE1: 0.26 L / s] are multimodal data features of highly correlated points, and ">" indicates the propagation path direction is from left to right.

[0121] [Weir WE1: 0.26 L / s] - [Inspection: Increased seepage flow in area C3 downstream of the dam, localized diffuse seepage] > [Video from camera 13#]. Here, [Weir WE1: 0.26 L / s] is the abnormal location, [Inspection: Increased seepage flow in area C3 downstream of the dam, localized diffuse seepage] and [Video from camera 13#] are the multimodal data features of the corresponding propagation path and highly correlated locations. The ">" indicates the propagation path direction is from left to right.

[0122] The subgraph transformation process is implemented in the following way:

[0123] 1. Based on human experience, the common and important features of anomaly correlation combination analysis graphs include: anomaly locations, propagation paths, and multimodal data characteristics.

[0124] 2. Following the expert's thinking, construct a natural language template and use string concatenation to transform these important features into natural language that humans can understand. From this natural language, summarize a general transformation template f(x), thus obtaining the feature-to-natural language template f(x).

[0125] 3. Then, all subsequent feature data can be mapped using this template f(x).

[0126] 4. Combine the transformed natural language with the corresponding prompts and knowledge to enable the large language model or knowledge platform to generate a corresponding graph description.

[0127] (2) Use multimodal data information to supplement features, for example:

[0128] Sensor data features (such as "the standard deviation of UP7 piezometer exceeds the threshold by 30% in the past 7 days") are obtained through API interfaces and associated with information such as image / text summaries (such as "the video inspection of the C3 leakage area shows surface diffusion") to form structured large model prompt words.

[0129] (3) Using a pre-trained large language model with expertise in water conservancy, combined with the context of the prompts provided in (1) and (2) and the user's questions, a multi-dimensional analysis report conclusion is formed, such as from causal reasoning of abnormal phenomena to quantitative conclusions of risk assessment. The steps are as follows:

[0130] 1. User question: Based on the current anomaly subgraph, what are the possible causes of the UP17 seepage gauge in section B3 of the dam?

[0131] 2. Forming the context of the prompt words (the following is a pseudocode representation of the prompt word context):

[0132] # You are a professional water conservancy expert. Based on the provided water conservancy knowledge, you can answer users' questions and generate professional reports.

[0133] ## Graph Data

[0134] Data source: (1)

[0135] ## Multimodal Data

[0136] Data source: (2)

[0137] Question: Based on the current anomaly subgraph, what are the possible causes of the UP17 seepage gauge in section B3 of the dam?

[0138] Output:

[0139] 1) The poor seepage prevention effect of the materials is caused by the poor geological conditions of the foundation of dam section No. 3 (confidence level 85%).

[0140] 2) High water levels cause seepage channels to form inside the dam;

[0141] 3) Instrument malfunction (requires on-site verification, probability <5%) ......

[0143] The built-in risk assessment algorithm is invoked to output a quantitative conclusion: Based on the current anomaly correlation, the system assesses the risk level as Level III, indicating a risk of leakage. It is recommended to prioritize the inspection of the dam foundation seepage prevention body and drainage system, while increasing the frequency of seepage monitoring and manual inspections behind the dam. Simultaneously, close monitoring of the measuring weir WE1 and the surrounding area should be maintained, and video inspections should be carried out using the nearby camera #13. Any increase in leakage should be reported and dealt with promptly.

[0144] Step 6: Interactive Visualization

[0145] The combined analysis charts and multimodal analysis conclusions are presented in a visual format and interactive functions are provided. Users can dynamically adjust the combination of analysis charts, view detailed information and export reports. It also supports calling the knowledge platform to query robots to refer to similar historical cases, providing new decision support for dam safety monitoring and analysis.

[0146] This invention also provides a dam safety monitoring data analysis system based on multimodal knowledge graphs and large language models, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the dam safety monitoring data analysis method based on multimodal knowledge graphs and large language models.

[0147] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A method for analyzing dam safety monitoring data based on multimodal knowledge graphs and large language models, characterized in that, Includes the following steps: Step 1: Collect multimodal data for dam monitoring and preprocess the multimodal data; Step 2: Construct a dam safety analysis knowledge graph using a multimodal knowledge graph construction method for water conservancy objects; Step 3: Use the intelligent agent to extract multimodal features, update the knowledge graph based on incremental data, use the intelligent positioning function to find the associated objects of the detected abnormal signals and obtain the information of the associated objects, and apply the dam analysis model to perform association analysis. Step 4: Based on the correlation analysis results obtained in Step 3, generate an abnormal correlation subgraph, which includes: abnormal points and correlation points; Step 5: Transform the abnormal correlation subgraph generated in Step 4 into a graph description language for input into the large language model. Use multimodal data information to supplement features and form structured large model prompt words. Use a pre-trained large language model with water conservancy expertise, combined with prompt words and user questions, to form a multidimensional analysis report conclusion.

2. The dam safety monitoring data analysis method based on multimodal knowledge graph and large language model according to claim 1, characterized in that, In step 1, the multimodal data includes: sensor monitoring data, images, videos, and text logs. The preprocessing includes: cleaning, denoising, and standardizing the data.

3. The dam safety monitoring data analysis method based on multimodal knowledge graph and large language model according to claim 1, characterized in that, In step 3, the specific steps for achieving intelligent positioning are as follows: First, the various ontologies involved in the water conservancy object and the relationships between them are constructed. Then, location information is added to the attributes of each ontology to build a knowledge graph data. Next, when a query is input, the entity name and the distance range to be queried are extracted from the query, and the distance between the entity and other water conservancy objects is calculated. Finally, entities within the range that meet the conditions are filtered out, and the connection relationship between the target entity and these entities is obtained. Finally, the entities and the connection relationship between entities are presented on the interface.

4. The dam safety monitoring data analysis method based on multimodal knowledge graph and large language model according to claim 1, characterized in that, In step 5, the multimodal data information includes: sensor data features obtained through API interfaces, and associated with image / text summaries.

5. The dam safety monitoring data analysis method based on multimodal knowledge graph and large language model according to claim 1, characterized in that, In step 5, the transformation of the anomaly association subgraph generated in step 4 into a graph description language for input into the large language model specifically includes: summarizing the common important features of the anomaly association combination analysis graph, transforming the important features into natural language, summarizing the common transformation template f(x) from the natural language, using the transformation module to map the subsequent feature data into natural language, and combining the transformed natural language with the corresponding prompt words and knowledge to allow the large language model or knowledge platform to generate the corresponding graph description language.

6. The dam safety monitoring data analysis method based on multimodal knowledge graph and large language model according to claim 1, characterized in that, Also includes: Step 6: Present the combined analysis chart and multidimensional analysis report conclusions in a visual format, and provide interactive functions, allowing users to dynamically adjust the combination of analysis charts, view detailed information and export reports. It also supports calling the knowledge platform query robot to refer to similar historical cases.

7. A dam safety monitoring data analysis system based on multimodal knowledge graphs and large language models, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the dam safety monitoring data analysis method based on multimodal knowledge graph and large language model as described in any one of claims 1-6.

Images