Water conservancy project risk response decision recommendation method based on cooperation of multi-modal knowledge graph and large model

Abstract

The invention discloses a water conservancy project risk response decision recommendation method based on cooperation of a multi-modal knowledge graph and a large model, and the method comprises the following steps: completing the construction process of the knowledge graph through three integrated modules: a knowledge graph mining module, a knowledge graph construction module, and a knowledge graph optimization module; by combining the respective advantages of the multi-modal knowledge graph and the multi-modal large model and adopting a retrieval generation collaborative iteration mode, the large model fully knows knowledge in the related water conservancy project inspection field, a targeted and most suitable countermeasure recommendation scheme is provided for the current project risk, the scheme is used for assisting water conservancy project maintenance personnel to carry out risk repair, and the risk repair efficiency is improved. Therefore, safe and stable operation of hydraulic engineering is ensured. The reply of the large language model and the field of water conservancy inspection are inseparable.

Description

Technical Field

[0001] The present invention belongs to the technical field of water conservancy risk response, and specifically relates to a water conservancy project risk response decision-making recommendation method that collaborates a multi-modal knowledge graph and a large model. Background Technique

[0002] A large amount of risk multi-modal information is recorded in the water conservancy project inspection text. However, if engineering maintenance personnel search for relevant risk problems manually, it is a major flaw in terms of both efficiency and accuracy, which greatly reduces the utilization rate of the effective information in the water conservancy project inspection text. Therefore, in order to enable engineering maintenance personnel to quickly and accurately obtain corresponding risk response strategies and improve the efficiency of engineering maintenance personnel in solving engineering safety problems, it is crucial to design an intelligent and efficient risk response decision-making recommendation system. However, existing risk response decision-making recommendation systems based on single-modal data knowledge graphs transform unstructured text and semi-structured data into structured data, and the quality of the data has been improved to a certain extent. However, they mainly utilize the text modal information in the water conservancy project inspection data, and the image modal data has not been fully utilized. Moreover, most of the above-mentioned knowledge graph construction methods rely on manual work, and water conservancy field experts are required to analyze and sort out a large amount of data to clarify entities and relationships. This process is both time-consuming and dependent on professional knowledge. At the same time, the coverage of the knowledge graph is often insufficient. The engineering maintenance problems in the water conservancy field are complex and diverse. Relying solely on the knowledge contained in the knowledge graph for risk decision-making recommendation will result in knowledge overflow problems. Multi-modal large models (MMLMs) contain a vast amount of prior knowledge and can effectively alleviate knowledge overflow problems. However, when facing domain problem tasks, large models often have problems of noise and randomness, and the interpretability of the answers is relatively low. To address the above problems and enable engineering maintenance personnel to quickly and accurately obtain corresponding risk response strategies and improve the efficiency of engineering maintenance personnel in solving engineering safety problems. The present invention uses a method that collaborates a multi-modal knowledge graph and a multi-modal large model to conduct research on a water conservancy project risk response decision-making recommendation method. On the one hand, multi-modal large models have a vast amount of knowledge and context learning capabilities, and their processing of images is becoming more and more mature, which can effectively alleviate the knowledge overflow problem existing in simply relying on the knowledge graph. On the other hand, to solve the problem of the single modality of the knowledge graph and the time-consuming and laborious problem of constructing traditional knowledge graphs, the present invention uses an LLMs-based method to automatically generate the knowledge graph ontology schema and constructs a multi-modal knowledge graph in the water conservancy field for water conservancy knowledge storage. Finally, by combining the multi-modal knowledge graph and the multi-modal large model, taking the multi-modal knowledge graph as enhanced knowledge, and adopting a multi-round retrieval generation collaborative iteration method to generate the optimal answer for risk handling, the above-mentioned solutions can complement each other and give full play to the advantages of each module, so as to achieve high accuracy and interpretability in water conservancy project risk response decision-making recommendation.

[0003] Most traditional knowledge graph construction methods rely on manual labor. Experts in the water conservancy field need to analyze and sort out a large amount of data to clarify entities and relationships. This process is both time-consuming and relies on professional knowledge, with extremely low efficiency. In addition, manual annotation methods mainly focus on the relationships between entities of the same type and often ignore the relationships between heterogeneous entities, affecting the extraction accuracy.

[0004] The currently popular recommendation methods are divided into two types: one is the recommendation method based on the water conservancy inspection knowledge graph; the other is the recommendation method for water conservancy project risk response decision-making based on large models (such as Tongyi Qianwen, GPT-4, etc.).

[0005] The disadvantages of the recommendation method based on the water conservancy inspection knowledge graph are that on the one hand, the covered knowledge is limited; on the other hand, the existing data utilization is mainly based on traditional single-text knowledge graphs, and multi-modal data in inspection texts is not used. It cannot provide accurate risk response decision-making recommendations for the personnel in the project operation and maintenance department.

[0006] The disadvantages of directly using language models for water conservancy project risk response decision-making recommendations are: although the model contains a large amount of prior knowledge, when facing specific domain tasks, it will bring problems such as hallucinations and poor interpretability; secondly, language models cannot analyze multi-modal data (such as images) and cannot make full use of inspection data. Summary of the Invention

[0007] To solve the above problems, the present invention provides a water conservancy project risk response decision-making recommendation method that synergizes a multi-modal knowledge graph and a large model. By using a knowledge graph ontology schema automatic construction method based on LLMs and constructing a multi-modal knowledge graph according to the constructed ontology schema; synergizing the water conservancy inspection multi-modal knowledge graph and the multi-modal large model for decision-making recommendations, fully complementing each other's advantages to solve the disadvantages.

[0008] The purpose of the present invention is achieved in the following manner: A water conservancy project risk response decision-making recommendation method that synergizes a multi-modal knowledge graph and a large model specifically includes the following steps: S1: Using thought framework prompts to assist the multi-modal large model in identifying entities and types within the water conservancy field and constructing a knowledge graph discovery module; The construction steps of the knowledge graph discovery module are as follows: First, using thought chain prompts to assist the multi-modal large model in identifying entities and types within the water conservancy field, that is Figure 1 the inspection text in Figure 2 and entity extraction and relationship extraction in Figure 2Entity type tags and relationship type tags. The meaning of the predefined rules here is that the relationships identified by the model are detailed. Facing the complex system of the water conservancy field, some rules are set manually to aggregate some similar relationships, thereby reducing its complexity. For example, the relationships identified by the model include the existence of standard hidden dangers, the existence of document hidden dangers, etc. We will collectively call these hidden dangers the existence of risk hidden dangers. Corresponding rules are set for other relationships. Then, relying on the multi-modal large model, through multi-level analysis, various entity types and relationship types are respectively fused into abstract high-dimensional types, that is Figure 2 Entity type fusion and relationship type fusion. At the same time, drawing on the idea of a fully connected graph, the water conservancy project entities and relationship types are exhaustively combined, and through entity-relationship joint extraction and dual-dimensional analysis, the type capture ability is improved, and potential meta-triples are screened out to the greatest extent to realize the automatic generation of the knowledge graph ontology concept model, that is Figure 2 Full-connected knowledge graph Schema generation -> Schema containing all potential meta-triples; S2: Use the risk local image-text pairs annotated by domain experts to train the YOLO model so that it can identify the local risk positions in the images. First: Assist in the construction of the multi-modal knowledge graph; Second: Assist in image knowledge retrieval during decision-making generation to provide more accurate decision-making schemes; that is Figure 1 The YOLO extraction module on the left; S3: Combine the generated Schema, use the multi-modal large model to extract instance triples from the inspection text, and through the YOLO model trained by domain knowledge, extract the images of the local risk parts to construct the multi-modal knowledge graph of the water conservancy field, that is Figure 3 Knowledge graph construction module; The construction method of the knowledge graph construction module is as follows: Based on the generated Schema, perform entity extraction, relationship extraction, and relationship set extraction of water conservancy project instances to construct a knowledge graph initially containing potential instance triples; S4: Count the number of triples of each instance corresponding to the meta-triple, delete the ones with lower frequencies, and complete the optimization of the knowledge graph, that is Figure 4 Knowledge graph optimization module, Figure 1 The knowledge graph optimization module on the left; The construction method of the knowledge graph optimization module is as follows: Use the frequency statistics method to remove suspicious and invalid instances, that is, the above-mentioned instances with lower frequencies, as Figure 4 shown in the "unreliable knowledge graph" in, improving the accuracy and reliability of the knowledge graph. This process realizes the automatic generation of the knowledge graph Schema and ensures the efficient construction of the knowledge graph; S5: Input the images and text instructions taken at the project site into the multi-modal large model, such as Figure 1As shown in the image risk identification in [reference], the optimized multi-modal knowledge graph in combination with step S4 collaborates with the multi-modal large model. By adopting the method of retrieval and generation collaborative iteration, the multi-modal large model can fully understand the knowledge in the field of water conservancy project inspection, and provide a targeted and most suitable recommendation scheme for the current project risks, that is, the generation of decisions. That is Figure 5 Knowledge enhancement framework; For the above-mentioned method for recommending water conservancy project risk response decisions through the collaboration of the multi-modal knowledge graph and the large model, the construction of the discovery module of the S1 knowledge graph is divided into six units: entity extraction, entity type annotation, entity type fusion, relationship extraction, relationship type annotation, and relationship type fusion. It deeply analyzes water conservancy project entities and relationship types from the seed text sequence, and combines the idea of a fully connected graph for extensive combination. Finally, it automatically generates a Schema containing all potential meta triples, enhancing the intelligence and efficiency of knowledge graph construction.

[0009] For the above-mentioned method for recommending water conservancy project risk response decisions through the collaboration of the multi-modal knowledge graph and the large model, in step S5, the risk location is captured from the images and text instructions taken at the project site through the YOLOv8 intelligent recognition technology. First, based on the keywords in the problem description, the first-round generation and retrieval interaction is carried out with the text knowledge in the multi-modal knowledge graph. Then, based on the image knowledge of the knowledge graph corresponding to the text knowledge retrieved in the first round, the second-round generation and retrieval interaction is carried out, that is, a detailed comparison is made with the image knowledge of the risk local part map identified by YOLOv8 to perform precise matching of local risk features. According to the knowledge obtained from the two rounds of iteration, the risk entity knowledge that best matches the features of the images taken at the project site is determined. The multi-modal large model will make full use of its powerful analysis ability, combined with the specific marked position information of the images uploaded from the project site and the enhanced knowledge continuously accumulated and optimized during the first two rounds of iteration, to conduct in-depth logical reasoning and risk assessment, and finally formulate a risk response decision that is both accurate and efficient. Figure 1 The image risk identification in [reference] serves for the similarity comparison of the second-round local risk map.

[0010] Compared with the prior art, the present invention has the following technical effects: (1) To alleviate the time-consuming and laborious problems brought about by traditional knowledge graph construction methods, the present invention uses an automatic construction method for knowledge graph ontology patterns based on LLMs, and constructs a multimodal knowledge graph according to the constructed ontology patterns; (2) To alleviate the disadvantages of directly using a water conservancy inspection knowledge graph or a large model to recommend risk response decisions for water conservancy projects, the present invention synergistically combines the multimodal water conservancy inspection knowledge graph and the multimodal large model for decision-making recommendations, fully complementing each other's advantages to address their disadvantages. In addition, to enable the model to focus on local risk positions in images, regarding the local position annotation problem involved, the present invention uses a trained YOLOv8 model for intelligent annotation. The present invention combines the respective advantages of the multimodal knowledge graph and the multimodal large model, and adopts a retrieval-generation collaborative iteration method, enabling the large model to fully understand the knowledge in the relevant water conservancy project inspection field, providing a targeted and most suitable response measure recommendation plan for the current project risks, and using this plan to assist water conservancy project maintenance personnel in risk repair, thereby ensuring the safe and stable operation of water conservancy projects, making the responses of large language models closely related to the water conservancy inspection field. Description of the Drawings

[0011] Figure 1 is a schematic flow chart of the present invention.

[0012] Figure 2 is a flow chart of the knowledge graph discovery module of the present invention.

[0013] Figure 3 is a flow chart of the knowledge graph construction module of the present invention.

[0014] Figure 4 is a flow chart of the knowledge graph optimization module of the present invention.

[0015] Figure 5 is a knowledge enhancement framework diagram of the present invention.

[0016] Figure 6 is an iterative retrieval-generation collaborative framework diagram of an embodiment of the present invention. Detailed Description of the Invention

[0017] The following combines Figure 1-6 to describe in detail the specific implementation method of the present invention.

[0018] A method for recommending risk response decisions for water conservancy projects by synergistically combining a multimodal knowledge graph and a large model specifically includes the following steps: S1: Using thought framework prompts to assist the multimodal large model in identifying entities and types in the water conservancy field, and constructing a knowledge graph discovery module, such as Figure 1 the knowledge graph discovery module shown. The construction steps of the knowledge graph discovery module are as follows: First, use chain-of-thought prompting to assist the multi-modal large model in identifying entities and relationships in the inspection texts in the water conservancy field, that is Figure 2 entity extraction and relationship extraction in Figure 2 , and explore entity types and relationship types through the multi-modal large model and predefined rules respectively, that is Figure 2 entity type marking and relationship type marking. The meaning of the predefined rules here is: The relationships identified by the model are detailed. Facing the complex system of the water conservancy field, some rules are set manually to aggregate some similar relationships, thereby reducing its complexity. For example: The relationships identified by the model include the existence of standard hidden dangers, the existence of document hidden dangers, etc. We will collectively call these hidden dangers the existence of risk hidden dangers. Corresponding rules are set for other relationships as well. Then rely on the multi-modal large model to fuse various entity types and relationship types into abstract high-dimensional types through multi-level analysis, that is Figure 2 entity type fusion and relationship type fusion. At the same time, draw on the idea of a fully connected graph, exhaustively combine water conservancy project entities and relationship types, and screen out potential meta triples to the greatest extent to achieve the automatic generation of the knowledge graph ontology concept model, that is Full connection knowledge graph Schema generation -> Schema containing all potential meta triples; Figure 1 S2: Train the YOLO model through the risk local image-text pairs annotated by domain experts so that it can identify the local risk positions in the images. First: Assist in the construction of the multi-modal knowledge graph; Second: Assist in image knowledge retrieval during decision-making generation to provide more accurate decision-making plans; that is The YOLO extraction module on the left; Figure 3 S3: Combine the generated Schema, use the multi-modal large model to extract instance triples from the inspection texts, and extract local risk part images through the YOLO model trained by domain knowledge to construct a multi-modal knowledge graph in the water conservancy field, that is Figure 1 The knowledge graph construction module; The construction method of the knowledge graph construction module is as follows: Based on the generated Schema, perform entity extraction, relationship extraction, and relationship set extraction of water conservancy project instances to construct a knowledge graph initially containing potential instance triples; S4: Count the number of triples of instances corresponding to each meta triple, delete the ones with lower frequencies, and complete the optimization of the knowledge graph, that is Figure 4 The knowledge graph optimization module; Figure 1 The knowledge graph optimization module on the left; The construction method of the knowledge graph optimization module is as follows: Use the frequency statistics method to remove suspicious and invalid instances, such as Figure 2The "Unreliable Knowledge Graph" shown improves the accuracy and reliability of the knowledge graph. This process realizes the automatic generation of the knowledge graph Schema, ensuring the efficient construction of the knowledge graph; S5: Input the images taken at the engineering site and the text instructions into a multi-modal large model, such as Figure 1 As shown in the image risk identification in, in combination with the optimized multi-modal knowledge graph in step S4, the multi-modal large model is coordinated, and in a way of retrieval and generation collaborative iteration, the multi-modal large model fully understands the knowledge in the relevant water conservancy project inspection field, and provides a targeted and most suitable response measure recommendation plan for the current project risk, that is, the generation of decision-making. That is Figure 5 The knowledge enhancement framework, that is Figure 1 The retrieval and generation iteration on the right.

[0019] For the water conservancy project risk response decision recommendation method that collaborates the multi-modal knowledge graph and the large model described in the present invention, the construction of the discovery module of the S1 knowledge graph is divided into six units: entity extraction, entity type annotation, entity type fusion, relationship extraction, relationship type annotation, and relationship type fusion. It deeply analyzes the water conservancy project entities and relationship types from the seed text sequence, and combines the idea of a fully connected graph for extensive combination, and finally automatically generates a Schema containing all potential meta triples, enhancing the intelligence and efficiency of the knowledge graph.

[0020] For the water conservancy project risk response decision recommendation method that collaborates the multi-modal knowledge graph and the large model described in the present invention, in step S5, the risk location is captured from the images taken at the engineering site and the text instructions through the YOLOv8 intelligent recognition technology. First, according to the keywords in the problem description, in the first round, a generation retrieval interaction is carried out with the text knowledge in the multi-modal knowledge graph. Then, based on the image knowledge of the knowledge graph corresponding to the text knowledge retrieved in the first round, a second-round generation retrieval interaction is carried out, that is, a detailed comparison is made with the image knowledge of the risk local part recognized by YOLOv8, and an accurate matching of local risk features is carried out. According to the knowledge of two rounds of iteration, the risk entity knowledge that best matches the features of the images taken at the engineering site is determined. The multi-modal large model will make full use of its powerful analysis ability, combined with the specific marked position information of the images uploaded at the engineering site and the enhanced knowledge continuously accumulated and optimized in the first two rounds of iteration, to conduct in-depth logical reasoning and risk assessment, and finally formulate a risk response decision that is both accurate and efficient. Figure 1 The image risk identification in is for the similarity comparison of the second-round local risk map.

[0021] Improvements of the present invention: (1) To alleviate the time-consuming and laborious problems brought by traditional knowledge graph construction methods, the present invention uses an automatic construction method for the ontology schema of the knowledge graph based on multimodal large language models (LLMs), and constructs a multimodal knowledge graph according to the constructed ontology schema; (2) To alleviate the disadvantages of directly using the water conservancy inspection knowledge graph or large language models to recommend risk response decisions for water conservancy projects, the present invention synergistically combines the water conservancy inspection multimodal knowledge graph and multimodal large language models for decision recommendation, fully complementing each other's advantages to solve the disadvantages. In addition, to enable the model to focus on local risk positions in images, the present invention uses the open-source multimodal large language model VIP-LLaVA that can focus on local positions. For the local position annotation problem involved, the present invention uses the trained YOLOv8 model for intelligent annotation. Specifically: First, to break through the existing limitations, a discovery-construction-optimization framework is designed to optimize the knowledge graph construction process. Specifically, first, chain-of-thought prompting is used to assist LLMs in identifying entities and relationships in the water conservancy field, and entity types and relationship types are explored relying on the model and predefined rules respectively. At the same time, drawing on the idea of a fully connected graph, the water conservancy project entities and relationship types are exhaustively combined to improve the type capture ability, and potential meta triples are screened out to the greatest extent to realize the automatic generation of the ontology concept model of the knowledge graph. Combining the generated Schema, instance triples are extracted from the inspection text with the help of LLMs to construct a multimodal knowledge graph in the water conservancy field. An optimization mechanism is designed to ensure the accuracy of the graph. The number of triples corresponding to each meta triple instance is counted, and the ones with lower frequencies are deleted. Second, to alleviate the problems of limited coverage of risk knowledge and imperfect utilization of multimodal data that occur by relying solely on the knowledge graph, and the lack of domain knowledge when relying solely on large language models for recommendation. The present invention combines the respective advantages of the multimodal knowledge graph and multimodal large language models, and adopts a retrieval-generation collaborative iteration method to enable the large language model to fully understand the relevant water conservancy project inspection field knowledge, and provide a targeted and most suitable response measure recommendation plan for the current project risk. This plan is used to assist water conservancy project maintenance personnel in risk repair, so as to ensure the safe and stable operation of water conservancy projects. Make the responses of the large language model inseparable from the water conservancy inspection field.

[0022] Detailed description of the automated construction of the knowledge graph ontology schema: The present invention completes the construction process of the knowledge graph through three integrated modules: the knowledge graph exploration module (KGExploration, KE), the knowledge graph construction module (KG Building, KB), and the knowledge graph optimization module (KGOptimization, KO). KE( Figure 2) It is divided into six units: entity extraction, entity type annotation, entity type fusion, relation extraction, relation type annotation, and relation type fusion to deeply analyze water conservancy project entities and relation types from the seed text sequence, and conduct extensive combinations in combination with the idea of a fully connected graph. Finally, a Schema containing all potential meta triples is automatically generated to enhance the practicality of the knowledge graph. The KB module ( Figure 3 ) Based on the generated Schema, it conducts entity extraction, relation extraction, and relation set extraction of water conservancy project instances to construct a knowledge graph initially containing potential instance triples. The KO module ( Figure 4 ) Adopts a frequency statistics method to remove suspicious and invalid instances, improving the accuracy and reliability of the knowledge graph. This process realizes the automatic generation of the knowledge graph Schema and ensures the efficient construction of the knowledge graph.

[0023] Details of the recommended method description: In the actual operation of engineering operation and maintenance, the operation and maintenance team will input the photos taken at the engineering site and detailed text instructions into the intelligent model. The model will first carefully analyze the text content and accurately capture the key engineering part information therein. Subsequently, it uses efficient Cypher query statements to search for entries similar to this key part in the knowledge graph database and extracts the countermeasures closely related to the risks of these similar parts, providing strong support for generating a preliminary recommended solution for the model. After obtaining the information of multiple similar parts, the model does not stop there, but starts the Cypher query statement again, delves into the knowledge graph, and searches for risk event graph entities closely connected to these risk part entities to further enrich its decision-making basis. At this time, the visual processor in the multimodal large model plays a crucial role. It will conduct in-depth visual analysis of the knowledge initially screened by the model, especially using the CLIP (Contrastive Language-Image Pretraining) technology to carefully compare the retrieved risk part images with the risk positions captured from the uploaded images through the YOLOv8 intelligent recognition technology, and conduct precise matching of local risk features, so as to determine the entity knowledge that best matches the local features of the currently uploaded image. On this basis, the MLLM will make full use of its powerful analysis ability, combine the specific marked position information of the uploaded image, and the enhanced knowledge continuously accumulated and optimized in the previous two rounds of iteration processes, conduct in-depth logical reasoning and risk assessment, and finally formulate a precise and efficient risk response decision, providing strong technical support and decision-making reference for the engineering operation and maintenance team.

[0024] The above are only the preferred embodiments of the present invention. It should be noted that for those skilled in the art, without departing from the overall concept of the present invention, several changes and improvements can still be made, and these should also be regarded as the protection scope of the present invention.

Claims

1. A water conservancy project risk response decision recommendation method based on multimodal knowledge graph and large model collaboration, characterized by: The specific steps include: S1: Using the thinking framework prompts, assist the multimodal large model to identify entities and types in the inspection text in the water conservancy field, and build a knowledge graph mining module; The construction steps of the knowledge graph mining module are as follows: first, using the thinking chain prompts to assist the multimodal big model to identify the entities and relationships in the inspection text in the water conservancy field, and respectively explore the entity type and relationship type through the multimodal big model and predefined rules, and then rely on the multimodal big model to integrate multiple entity types and relationship types into abstract high-dimensional types through multi-level analysis. At the same time, drawing on the idea of ​​fully connected graphs, exhaustively combining water conservancy project entities and relationship types, and maximally screening out potential meta-triplets, so as to realize the automatic generation of the knowledge graph ontology concept model; S2: The YOLO model is trained using risky local image-text pairs annotated by domain experts to enable it to identify local risk locations in images; S3: Combined with the generated Schema, the instance triples are extracted from the inspection text with the help of the multimodal large model, and the risk local area images are extracted through the YOLO model that has been trained with domain knowledge to construct a multimodal knowledge graph in the water conservancy field, namely the knowledge graph construction module in Figure 3; The method for constructing the knowledge graph construction module is as follows: based on the generated Schema, entity extraction, relationship extraction and relationship set extraction of water conservancy project instances are performed to construct a preliminary knowledge graph containing potential instance triples; S4: Count the number of triples corresponding to each instance of the triple, delete the ones with lower frequency, and complete the optimization of the knowledge graph; The construction method of the knowledge graph optimization module is as follows: the frequency statistics method is used to remove suspicious and invalid instances, thereby improving the accuracy and reliability of the knowledge graph. This process realizes the automatic generation of the knowledge graph Schema and ensures the efficient construction of the knowledge graph; S5: Input the images taken at the project site and the text instructions into the multimodal big model together, combine the multimodal knowledge graph optimized in step S4 with the multimodal big model, and adopt the retrieval generation collaborative iteration method to enable the multimodal big model to fully understand the relevant water conservancy project inspection field knowledge, and provide targeted and most appropriate response measures for the current project risks. Recommended solutions, that is, decision generation.

2. The water conservancy project risk response decision recommendation method based on the multimodal knowledge graph and large model collaboration according to claim 1 is characterized by: The construction of the S1 knowledge graph mining module is divided into six units: entity extraction, entity type labeling, entity type fusion, relationship extraction, relationship type labeling, and relationship type fusion. It deeply analyzes the water conservancy project entities and relationship types from the seed text sequence, combines them extensively with the idea of ​​a fully connected graph, and finally automatically generates a Schema containing all potential meta-triplets, thereby enhancing the intelligence and efficiency of knowledge graph construction.

3. The water conservancy project risk response decision recommendation method based on the multimodal knowledge graph and large model collaboration according to claim 2 is characterized by: In step S5, the risk location is captured from the images and text instructions taken at the engineering site through the YOLOv8 intelligent recognition technology. First, based on the keywords in the problem description, the first round of generation and retrieval interaction is performed with the text-based knowledge in the multimodal knowledge graph. Then, the second round of generation and retrieval interaction is performed based on the image knowledge of the knowledge graph corresponding to the text knowledge retrieved in the first round, that is, a detailed comparison is made with the risk local part map knowledge identified by YOLOv8, and the local risk characteristics are accurately matched. Based on the knowledge of the two rounds of iterations, the risk entity knowledge that best matches the characteristics of the images taken at the engineering site is determined. The multimodal large model will make full use of its powerful analysis capabilities, combined with the specific mark location information of the images taken and uploaded at the engineering site, and the enhanced knowledge continuously accumulated and optimized during the first two rounds of iterations, to conduct in-depth logical reasoning and risk assessment, and finally formulate risk response decisions that are both accurate and efficient.

Images