A method and system for constructing and operating a multi-modal large model for reservoir dam safety

Abstract

The present invention provides a method and system for constructing and operating a multi-modal large model for reservoir dam safety. The method includes the following steps: constructing a training set: a single sample of the training set includes; multi-modal data as model input: sensor data deployed on the dam, historical hidden danger records, engineering technical documents formed in each stage of the dam's planning, design, construction, and operation, information resources related to dam safety, geometric and material information of the BIM model, timestamp and spatial location information; and the probability value of whether a hidden danger event occurs as the training label; training a multi-modal pre-trained model based on the Transformer architecture using the training set; performing real-time prediction and early warning on dam safety hidden dangers; calculating the actual impact degree, comprehensive risk score, dynamically set early warning threshold, and emergency measure suggestions of the hidden danger event according to the probability value of whether the hidden danger event occurs output by the model. The present invention comprehensively improves the scientificity and effectiveness of dam safety monitoring.

Description

Technical Field

[0001] The present invention belongs to the technical field of dam safety protection, and particularly relates to a method and system for constructing and operating a multi-modal large model for reservoir dam safety. Background Art

[0002] As an important part of water conservancy projects, the safety of dams directly affects the lives and property of residents in downstream areas and the stability of the ecological environment. Therefore, the health monitoring and safety assessment of dams have always been important topics in the global engineering and scientific research fields. With the increase in the service life of dams, frequent natural disasters, and extreme environments brought about by climate change, the need for health monitoring of dam structures has become even more urgent. How to conduct safety monitoring in real time and efficiently in a complex and changing environment has become a key challenge.

[0003] Traditional dam safety monitoring mainly relies on manual inspections and single-modal sensors, and these methods have obvious deficiencies in terms of real-time performance, accuracy, and comprehensiveness. Manual inspections are inefficient, rely on experience, and it is difficult to cover all potential hazards. Single-modal monitoring, such as strain sensors, can only obtain data on a specific aspect of the dam and cannot provide an overall health assessment. For example, strain sensors can detect abnormalities but cannot identify problems such as surface cracks. The fusion and analysis of multi-modal data are important ways to improve the comprehensiveness of monitoring, but the data heterogeneity of different sensor types is high, and the fusion analysis is difficult, and it is often impossible to effectively reveal comprehensive risks.

[0004] Existing dam monitoring systems also have limitations in intelligent analysis and decision-making capabilities. Most systems rely on fixed thresholds to trigger early warnings, lack flexibility, and cannot dynamically respond to complex environmental changes. In addition, existing early warning systems usually cannot deeply analyze the causes of potential hazards or provide effective countermeasures, making it difficult for management personnel to make scientific decisions in a timely manner.

[0005] The lack of historical data and knowledge bases further limits the effectiveness of existing systems. Existing systems mostly rely on real-time data, lack the correlation analysis with historical potential hazard records and maintenance and reinforcement data, and cannot make full use of past experience for intelligent auxiliary decision-making. Historical information during dam operation is often of great significance for predicting potential risks, and the isolated analysis methods of existing systems tend to ignore the value of this information.

[0006] The lack of real-time monitoring and dynamic prediction capabilities is also a shortcoming of current dam monitoring technologies. In the face of climate change and natural disasters, the state of dams will evolve dynamically over time, and existing static monitoring methods are difficult to identify structural potential hazards in a timely manner and issue effective early warnings. Summary of the Invention

[0007] The objective of the present invention is to address the deficiencies existing in the above-mentioned background technology, and to provide a method and system for constructing and operating a multi-modal large model for reservoir dam safety, so as to comprehensively improve the scientificity and effectiveness of dam safety monitoring.

[0008] The technical solution adopted by the present invention is: a method for constructing and operating a multi-modal large model for reservoir dam safety, including the following steps:

[0009] Construct a training set: Each single sample of the training set includes: multi-modal data as the model input: sensor data deployed on the dam, historical hidden danger records, engineering technical documents formed in each stage of the dam's planning, design, construction, and operation, information resources related to dam safety, geometric and material information of the BIM model, timestamp and spatial location information; and whether a hidden danger event occurs and the probability value of the occurrence of the hidden danger event as the training label.

[0010] Use the training set to train a multi-modal pre-training model based on the Transformer architecture.

[0011] Use the real-time collected data to perform real-time prediction and early warning of dam safety hidden dangers based on the trained model.

[0012] Calculate the actual impact degree of the hidden danger event, the comprehensive risk score, the early warning threshold of the sensors associated with the hidden danger event, and the emergency measure suggestions according to the probability value of whether the hidden danger event occurs output by the model.

[0013] In the above technical solution, the data is preprocessed before being used to construct the training set: the original data of the collected training samples is cleaned to remove abnormal data and invalid data; the data is labeled in a unified format, and the data is sorted into structured data suitable for model training; the time alignment of the image and vibration data is achieved through the timestamp and the unified processing of the spatial coordinates.

[0014] In the above technical solution, the sensors deployed on the dam include but are not limited to: physical quantity sensors, vibration sensors, visual sensor data, audio sensors, and environmental sensors.

[0015] In the above technical solution, during the model training process, the binary cross-entropy loss function, the gradient descent algorithm, and L2 regularization are used to optimize the model parameters.

[0016] In the above technical solution, before the training and use of the multi-modal pre-training model based on the Transformer architecture, the data of each modality input is preprocessed through an independent encoder, and the features of each modality are fused through the multi-head self-attention mechanism of the Transformer and used as the model input.

[0017] In the above technical solution, the actual impact degree C of the potential hazard event is calculated by the following formula impact :

[0018] ;

[0019] where, I i represents the impact degree of the i-th type of potential hazard event on the dam, is the occurrence probability of this type of potential hazard event, and n represents the number of potential hazard event types.

[0020] In the above technical solution, the comprehensive risk score R is calculated by the following formula:

[0021]

[0022] where, α is the weight of the hazard occurrence probability P failure indicating its relative importance in risk assessment, and β is the weight of the hazard impact degree, indicating its contribution to the comprehensive score.

[0023] In the above technical solution, the warning threshold of the sensor associated with the potential hazard event is comprehensively adjusted according to the following principles and their corresponding weights:

[0024] When the occurrence probability of any type of potential hazard is greater than its set upper threshold, the warning threshold of the sensor associated with this potential hazard is adjusted accordingly according to the amplitude by which the probability is greater than its set upper threshold, so as to improve the sensitivity of the sensor; when the occurrence probability of any type of potential hazard is less than its set lower threshold, the warning threshold of the sensor associated with this potential hazard is adjusted accordingly according to the amplitude by which the probability is less than its set lower threshold, so as to reduce the sensitivity of the sensor;

[0025] When the information resources related to the dam safety input include weather forecasts of a set type and degree, the warning threshold of the sensor associated with this weather forecast is adjusted according to the weather forecast, so as to improve the sensitivity of the sensor;

[0026] The current warning threshold of the sensor is adjusted according to the deviation between the historical warning threshold of the sensor and the historical potential hazard records;

[0027] The current warning threshold of the sensor is adjusted according to the abnormal fluctuation of the sensor.

[0028] In the above technical solution, the process of determining the emergency measure suggestions includes: when the occurrence probability of any type of potential hazard is greater than its set upper threshold, according to the technical standards and specifications included in the information resources related to the dam safety input and the historical maintenance and reinforcement records of the corresponding potential hazards in the engineering technical documents, the corresponding emergency measure suggestions for this potential hazard are obtained.

[0029] The present invention also provides a multi-modal large model construction and operation system for reservoir dam safety, including a training set construction module, a model training module, a prediction module, and a warning module:

[0030] The training set construction module is used to construct a training set: A single sample of the training set includes; multi-modal data as the model input: sensor data deployed on the dam, historical hidden danger records, engineering technical documents formed in each stage of the dam's planning, design, construction, and operation, information resources related to dam safety, geometric and material information of the BIM model, timestamp and spatial location information; and whether a hidden danger event occurs and the probability value of the hidden danger event occurring as the training label;

[0031] The model training module is used to train a multi-modal pre-trained model based on the Transformer architecture using the training set;

[0032] The prediction module is used to perform real-time prediction and warning of dam safety hidden dangers based on the trained model using real-time collected data;

[0033] The warning module is used to calculate the actual impact degree, comprehensive risk score, warning threshold of the sensors associated with the hidden danger event, and emergency measure suggestions according to the probability value of whether the hidden danger event occurs output by the model.

[0034] The beneficial effects of the present invention are: The present invention integrates multi-modal data sources (operation status data, historical records, technical documents, BIM models, etc.) to achieve a comprehensive perception of the dam's state throughout its life cycle. The breadth and depth of the data ensure that the model can accurately predict hidden danger events; through the multi-modal pre-trained model based on the Transformer architecture, the multi-head self-attention mechanism is used to efficiently capture the correlation between modalities, improving the accuracy of hidden danger prediction; the real-time prediction and warning capabilities ensure that potential safety hidden dangers of the dam can be responded to in a timely manner, providing reliable technical support for dam operation management; providing the actual impact degree, comprehensive risk score, and dynamic warning threshold provides a scientific basis for emergency response.

[0035] Furthermore, the present invention uses data cleaning and anomaly detection to remove invalid or interfering information, ensuring the high quality of the training data and the robustness of the model; unified formatting and time alignment processing improve the degree of data structuring and provide stable input for model training; the alignment of time and space information ensures the spatio-temporal consistency between multi-modal data and enhances the model's ability to capture complex hidden danger patterns.

[0036] Furthermore, the multi-modal data sources of the present invention cover the key monitoring data (physical quantities, vibrations, vision, audio, environment) during the operation of the dam, comprehensively reflecting the real-time state of the dam; the data of different types of sensors jointly provide multi-dimensional information sources, improving the reliability and comprehensiveness of the model to identify potential hazard events; the data input including that of environmental sensors takes into account the impact of external environmental changes on the dam safety, enhancing the adaptability of the system.

[0037] Furthermore, the present invention adopts the binary cross-entropy loss function to adapt to the classification problem of potential hazard prediction, ensuring the optimization of classification accuracy; the use of the gradient descent algorithm and L2 regularization improves the training efficiency of the model, avoids overfitting, and enhances the generalization ability of the model; through the combination of optimized algorithms, the stability and prediction ability of the model in complex data scenarios are ensured.

[0038] Furthermore, the independent encoding method for each modal data of the present invention retains the feature independence between modalities, ensuring that information will not be lost in the early stage; the multi-head self-attention mechanism of the Transformer architecture fuses multi-modal features, effectively capturing the high-order interactions between modalities and improving the accuracy of potential hazard prediction; the modular structure of the model is convenient for expansion and can adapt to future newly added data modalities or changing system requirements.

[0039] Furthermore, through the calculation of the impact degree of potential hazard events, the present invention quantifies the actual impact of potential hazard events, providing a basis for subsequent comprehensive risk scoring; the quantitative calculation of the impact degree provides a clear analysis basis for safety management, helping decision-makers formulate more scientific response strategies.

[0040] Furthermore, the comprehensive risk scoring of the present invention integrates the probability of potential hazard occurrence and the impact degree, providing a more comprehensive safety evaluation index; by introducing a weight factor, the weights of potential hazard probability and impact degree in risk assessment can be flexibly adjusted to meet the requirements of different scenarios; the provided comprehensive risk scoring lays a foundation for setting warning thresholds and suggesting emergency measures.

[0041] Furthermore, when the occurrence probability of a potential hazard event exceeds the set upper threshold, the present invention automatically increases the sensitivity of relevant sensors so as to detect potential safety hazards earlier and more accurately; when the occurrence probability of a potential hazard event is lower than the lower threshold, the sensitivity of the sensors is correspondingly decreased, thereby avoiding overreaction of the sensors and reducing unnecessary early warnings. When the sensors are associated with weather forecast data, the present invention can adjust the early warning threshold of the sensors according to weather changes, especially under extreme weather conditions, to give early warnings in advance and enhance the risk resistance ability of the dam. Based on the deviation between the historical early warning threshold of the sensors and the potential hazard records, as well as the abnormal fluctuations of the sensors, the present invention dynamically adjusts the current early warning threshold to avoid misjudgment or missed judgment caused by equipment failures or historical record deviations. By adjusting the sensitivity of the sensors in real time, the present invention can more accurately reflect the risk status of actual potential hazard events, avoid false alarms and missed alarms, and can comprehensively judge according to various aspects of information, timely adjust the working state of the equipment, and further improve the overall safety of the dam. Combining multi-modal data and external factors (such as weather forecasts, historical data, etc.), the early warning system can adaptively adjust under changing environmental conditions and improve the emergency response ability of the dam.

[0042] Furthermore, the present invention proposes a process for determining emergency measure suggestions through technical standards, engineering technical documents, and historical maintenance records. When the occurrence probability of a potential hazard event exceeds the set threshold, the system will provide standardized emergency measure suggestions for specific types of potential hazards according to the design standards, construction specifications, and operation requirements of the dam. This can ensure that the emergency measures comply with industry standards and ensure their effectiveness and feasibility. Based on the maintenance and reinforcement records of historical potential hazards, the system can extract the experience of solving similar problems in the past and provide specific solutions for the current potential hazards. The application of historical experience helps to avoid repeated mistakes and improve the efficiency and quality of emergency response. The present invention can automatically generate emergency measures according to the occurrence probability of real-time potential hazard events and relevant data, reduce manual intervention, and improve the speed of emergency response; by combining historical maintenance records and technical standards, the system can transform the accumulated experience and technical specifications into emergency response plans, enhancing the intelligent level and decision-making support ability of the system; timely and targeted emergency measure suggestions help to improve the safety protection ability of the dam and reduce the losses and risks after accidents occur.

[0043] Furthermore, the present invention can not only improve the intelligent level of the dam safety monitoring system, but also enhance its early warning and emergency response capabilities for potential hazard events. Through the dynamic adjustment of the early warning threshold of the sensors and the intelligent generation of emergency measures, it can effectively predict, evaluate, and handle the potential risks of the dam in a changing environment, thereby providing more reliable guarantees for the long-term safe operation of the dam. The application of these technical solutions can greatly reduce the probability of serious accidents occurring in the dam, improve its risk management level, and ultimately ensure the safety and social benefits of the dam. Brief Description of the Drawings

[0044] Figure 1 is the overall architecture diagram of the system of the present invention;

[0045] Figure 2 is the flow chart of multi-modal monitoring data acquisition and processing inside the dam;

[0046] Figure 3 is the flow chart of external resource data acquisition and processing based on web search;

[0047] Figure 4 is the flow chart of dam historical data retrieval based on RAG;

[0048] Figure 5 is the flow chart of dam hidden danger assessment based on BIM model, mathematical and physical models and their hybrid models;

[0049] Figure 6 is the flow chart of dam hidden danger propagation path analysis based on knowledge graph and graph database;

[0050] Figure 7 is the flow chart of model fine-tuning of the present invention;

[0051] Figure 8 is the flow chart of the dam safety comprehensive early warning and response mechanism of the present invention. Detailed Embodiment

[0052] The present invention will be further described in detail below in conjunction with the drawings and specific embodiments, which are convenient for clearly understanding the present invention, but they do not limit the present invention.

[0053] As Figure 1 shown, the present invention provides a method for constructing and operating a multi-modal large model for reservoir dam safety, including the following steps:

[0054] Construct a training set: Each single sample of the training set includes: multi-modal data as the input of the model: sensor data deployed on the dam, historical hidden danger records, engineering technical documents formed in each stage of the dam's planning, design, construction and operation, information resources related to dam safety, geometric and material information of the BIM model, time stamp and spatial position information; and whether the hidden danger event occurs and the probability value of the hidden danger event occurring as the training label;

[0055] Train a multi-modal pre-trained model based on the Transformer architecture using the training set;

[0056] Use the real-time collected data to perform real-time prediction and early warning of dam safety hidden dangers based on the trained model;

[0057] Calculate the actual impact degree, comprehensive risk score, warning threshold of the sensors associated with the potential hazard event, and emergency measure suggestions of the potential hazard event based on the probability value of whether the potential hazard event occurs output by the model.

[0058] The principle of the present invention will be further described below in conjunction with specific embodiments.

[0059] Embodiment 1: A multi-modal large model construction and operation system for reservoir dam safety

[0060] As Figure 1 shown, a multi-modal large model construction and operation system for reservoir dam safety includes the following modules:

[0061] Module 1, a real-time perception module for multi-modal internal monitoring data of the dam, which is used for collecting and processing multi-modal monitoring data inside the dam, obtaining real-time and historical operation status data, and generating a comprehensive real-time monitoring view of the dam's health status.

[0062] The monitoring data includes physical quantities (such as deformation, seepage, stress and strain, and temperature), images (such as photos, videos, and point clouds), vibrations (such as strong earthquake monitoring), audio frequencies (such as metal structure monitoring), environment (such as meteorology, earthquake, and hydrology), etc.

[0063] The real-time monitoring view of the dam's health status is a data visualization interface that comprehensively displays the current safety status and structural health of the dam. The real-time monitoring view of the dam's health status displays the following data:

[0064] Real-time data overview: Displays various real-time monitoring data of the dam, including physical quantities such as stress, displacement, temperature, and seepage.

[0065] Status indicators: Health indicators calculated from various monitoring data, such as safety factors and risk levels, to help quickly judge the overall health status of the dam.

[0066] Anomaly detection: Real-time displays abnormal data or warning signals detected by sensors to help management personnel timely identify potential safety hazards.

[0067] Visualization charts: Intuitively display data change trends in the form of graphs and charts, such as stress change curves and displacement distribution maps, to facilitate the identification of abnormal patterns. Regional heat maps: Display the health status of different parts of the dam through heat maps, where the color depth represents different risk levels or the degree of abnormality of monitoring data.

[0068] Historical comparison: Provides a comparison of historical monitoring data to help analyze the changes between the current state and the past state, identify long-term trends and potential problems.

[0069] Combining these information, the real-time monitoring view of the dam health status provides a clear and intuitive tool for managers to make quick and scientific decisions and ensure the safe operation of the dam.

[0070] Module 2: Real-time perception module for external resource data based on web search, which is used to obtain information resources related to dam safety globally through automated web crawler technology.

[0071] Specifically, through web search and data scraping, a large amount of external resource data is obtained; an inverted index and a B+ tree index are established for the external resource data; the index and parallel query technology are used to retrieve the external resource data to find the target data; the data sources and query results with high-frequency access are cached; the target data and the dam operation status data are weighted and fused, and the weights are dynamically adjusted according to the data characteristics to form a comprehensive information resource dataset.

[0072] The information resource data related to dam safety includes news reports on reservoir dam monitoring, safety assessment, monitoring and early warning, and repair and reinforcement globally, as well as the latest academic papers, patents, laws, regulations, and standards and specifications and other scientific and technological resources. These data are combined with multi-modal sensor data to enhance the assessment ability of the overall safety status of the dam. The introduction of external resource data improves the comprehensiveness of decision-making and provides rich background information and reference basis for the subsequent analysis module.

[0073] Module 3: RAG-based historical data retrieval module, which constructs a knowledge document library based on the engineering and technical documents formed in each stage of the dam's planning, design, construction, and operation; divides the long documents in the knowledge document library into text blocks; uses a text embedding model to convert the text blocks into digital vectors to capture the semantic information of the text and form a vector database; the vector database is used to retrieve the text blocks related to the query and provide the most relevant information.

[0074] The dam knowledge document library covers various important data types such as survey and design, construction records, dam daily operation data, hidden danger records, log records, operation and maintenance records of hydraulic gates and water turbine units, and dam repair and reinforcement records, covering detailed information in all aspects of dam construction, operation, maintenance, and repair.

[0075] To ensure that this information can be retrieved and utilized efficiently and accurately, in the first step of Module 3, the historical data is vectorized through the embedding technology, converting the text information into digital vectors for subsequent calculation and comparison. This vectorized representation not only retains the semantic information of the data but also improves the data processing efficiency.

[0076] During the retrieval process, after the user enters the query keywords, the system first vectorizes these keywords to generate a query vector. Then, the system uses the stored vectors in the vector database to find the historical data vectors that are closest to the query vector through similarity calculation (such as cosine similarity). This process can quickly identify the documents or records relevant to the user's needs.

[0077] To further improve the relevance of the retrieval results, the system then adopts the reranking technology. Based on the preliminary retrieval, this technology performs a secondary ranking on the results. By introducing more context information and features (such as the importance of the document, historical retrieval frequency, etc.), it optimizes the ranking order of the retrieval results.

[0078] Specifically, reranking may use a machine learning model to score the preliminary results, taking into account the relevance between the document and the query, the quality of the document, and the user's historical preferences, to ensure that the user can quickly find the information that best meets their needs when querying, thus providing strong support for subsequent risk assessment and decision-making.

[0079] This knowledge document library provides support for subsequent data analysis and at the same time provides historical context for the system to help analyze the current monitoring data more accurately. The historical data of the dam not only provides a key reference for the analysis of real-time monitoring data but also adds the necessary historical background to the assessment process, making the dam safety assessment more comprehensive and accurate.

[0080] Module Four: Construct a hidden danger assessment module based on BIM models, mathematical models, physical models, and mathematical-physical hybrid models, which is used to evaluate the structural strength and stability of the dam based on real-time monitoring data and historical data, and obtain the location and type of hidden dangers.

[0081] The mathematical models, physical models, and mathematical-physical hybrid models include finite element analysis models, fluid mechanics models, heat conduction models, and soil mechanics models.

[0082] The finite element analysis model is used to identify stress concentration areas through finite element analysis; the fluid mechanics model is used to identify areas where structural damage may occur due to excessive water pressure or water flow scouring through fluid mechanics analysis; the heat conduction model is used to identify the parts where temperature changes affect structural safety through thermal stress distribution analysis; the soil mechanics model is used to identify areas where settlement or landslides may occur through soil mechanics analysis.

[0083] Specifically, Module Four constructs the BIM model of the dam based on engineering and technical documents, extracts attribute data therefrom and assigns it to each structural unit; associates information such as historical operation status data, maintenance records, and stress history with the corresponding parts of the BIM model; based on the externally obtained information resources, incorporates the latest technical standards, specifications, laws and regulations, accident cases, and academic research into the BIM model, and updates the risk assessment parameters and hidden danger identification model;

[0084] Using the geometric and material information of the BIM model, perform finite element analysis, fluid mechanics analysis, heat conduction analysis, and soil mechanics analysis to simulate the response of the dam under various working conditions, and identify potential hidden danger areas and characteristics, including stress concentration areas, areas that may be subject to excessive water pressure or erosion, areas that may generate cracks due to excessive temperature stress, and foundation areas that may experience settlement and landslides;

[0085] Compare the identified hidden danger areas with the historical hidden danger records in the historical operation status data and engineering and technical documents to verify the accuracy of the model analysis results; classify the identified historical hidden dangers according to the historical hidden danger characteristics;

[0086] Highlight the identified hidden danger areas in the BIM model and mark the hidden danger characteristics; continuously update the BIM model and various analysis models with the acquisition of new operation data and information resources.

[0087] Module Four realizes the dynamic assessment of the dam safety hidden dangers by integrating the BIM (Building Information Modeling) model, mathematical model, physical model, and hybrid models of mathematical and physical models, combined with internal monitoring data, external resource data, and historical data. The core of this module is to integrate multiple models to provide comprehensive and scientific analysis results to support the safety management and decision-making of the dam. By integrating these models, the system can obtain the hidden dangers of the dam and the locations where the hidden dangers occur.

[0088] Module Five, a hidden danger propagation path analysis module based on knowledge graph and graph database, is used to construct a knowledge graph and a graph database, identify associated hidden dangers by analyzing the relevance between the dam structure nodes according to real-time monitoring data and historical data, and realize the analysis and early warning of the dam hidden danger propagation path to obtain associated hidden danger information and hidden danger propagation paths.

[0089] Specifically, Module Five defines each structural unit of the dam as a structural unit node and assigns attributes; defines the physical connections and mechanical relationships between structural units as structural connection edges and assigns attributes;

[0090] Establish the association relationship between nodes and edges: establish the edges between structural unit nodes according to the historical operation status data, engineering and technical documents, and geometric information in the BIM model;

[0091] Store the information of nodes and edges into a graph database; calculate the connectivity and importance scores of nodes by performing calculations on the nodes and edges in the graph database.

[0092] Based on the real-time collected dam operation status data and new external information resources, update the health status and risk levels of nodes in the knowledge graph, as well as the weights and influence probabilities of edges.

[0093] Utilize the results calculated from the updated node and edge attributes and connectivity to analyze the paths and probabilities of potential hazards spreading from one structural unit to another.

[0094] Based on the potential hazard spreading probabilities, construct a matrix representing the relevance of potential hazards between nodes; combine the potential hazard impact matrix and the graph database to calculate the shortest paths of potential hazards from the source node to other nodes; identify the nodes that have important impacts during the spread of potential hazards according to the importance scores of nodes and their roles in the spread of potential hazards.

[0095] Since the structure of the dam consists of multiple interconnected components, any abnormality in a single node may trigger a chain reaction, thereby affecting the safety of the entire structure. The knowledge graph represents the various structural units of the dam and their interrelationships in a graphical manner, enabling the system to clearly display the connections and influences between nodes. As the underlying support of the knowledge graph, the graph database can efficiently store and query complex structural information. By abstracting the structural units of the dam as nodes in the graph, the attributes of each node not only include basic physical information but also can integrate multi-dimensional data such as historical potential hazard records, detection data, and maintenance logs. This integration of information enables the system to quickly retrieve relevant nodes and analyze possible associated potential hazards when a potential hazard occurs.

[0096] In addition, by leveraging the powerful path calculation ability of the graph database, the system can achieve precise analysis of the paths of potential hazard spread. When an abnormality occurs in a certain node, the system can quickly calculate the other nodes that may be affected by this potential hazard and identify the shortest path of potential hazard spread.

[0097] By combining the knowledge graph and the graph database, not only can the structural relevance of the dam be systematically analyzed, but also potential associated potential hazards and their locations can be comprehensively identified, providing more reliable data support and decision-making basis for ensuring the safety of the dam. Through this multi-level analysis method, the dam monitoring system can achieve efficient linkage from early warning to decision-making, ensuring the overall health and safety of the dam.

[0098] Module Six: Pre-training and Fine-tuning Module of Multimodal Large Model. The pre-training of the multimodal large model is carried out by monitoring historical data, external data, historical materials, hidden danger locations and types, and hidden danger propagation paths, enabling it to learn to obtain risk assessment results, warning thresholds, and warning response mechanisms based on the real-time monitoring data, external resource data, dam historical materials, hidden danger locations and types, and dam hidden danger propagation paths. Then, the multimodal large model is fine-tuned in combination with the dam real-time monitoring data to achieve intelligent analysis and adaptive optimization of multimodal data.

[0099] After pre-training and fine-tuning the multimodal large language model, a multimodal large language model for dam safety monitoring and assessment is obtained.

[0100] Specifically, during the model training process, the binary cross-entropy loss function, gradient descent algorithm, and L2 regularization are used to optimize the model parameters.

[0101] Before training and using the multimodal pre-training model based on the Transformer architecture, the data of each input modality is preprocessed through an independent encoder, and the features of each modality are fused through the multi-head self-attention mechanism of the Transformer and used as the model input.

[0102] During the training process of the multimodal large language model, the construction of the dataset is the key to the success of the system. In the data preprocessing stage, data cleaning, denoising, and annotation are carried out to ensure the accuracy and consistency of the input data. Through high-quality data preprocessing, the training efficiency of the model is improved, and a solid foundation is laid for subsequent analysis and decision support.

[0103] To improve the model performance, this module uses the fine-tuning technology to fine-tune the trained multimodal large language model to make the model better adapt to the dam monitoring requirements.

[0104] Module Seven: Comprehensive Decision-making and Early Warning Module Based on Multimodal Large Model.

[0105] This module is used for dam safety monitoring and assessment. The pre-trained multimodal large language model is used for dam safety monitoring and assessment. By obtaining the real-time monitoring data, the probability of hidden danger occurrence, the impact degree of hidden danger, and the warning threshold are obtained, and then the risk assessment results and warning response measure suggestions are output.

[0106] The actual impact degree C of the hidden danger event is calculated using the following formula impact :

[0107] ;

[0108] where I i represents the impact degree of the i-th type of hidden danger event on the dam, is the occurrence probability of this type of hidden danger event, and n represents the number of hidden danger event types.

[0109] In the above technical solution, the comprehensive risk score R is calculated by the following formula:

[0110]

[0111] Among them, α is the weight of the occurrence probability P of the hidden danger failure (which can be obtained by weighted summation of the occurrence probabilities of various hidden dangers), indicating its relative importance in risk assessment, and β is the weight of the hidden danger impact degree, indicating its contribution to the comprehensive score.

[0112] Specifically, according to the following principles and their corresponding weights, the warning threshold of the sensor associated with the hidden danger event is comprehensively adjusted. The setting of the weight can be set according to human experience, and the final adjustment value is calculated according to the corresponding adjustment range and the corresponding weight of each principle:

[0113] When the occurrence probability of any type of hidden danger is greater than its set upper threshold, the warning threshold of the sensor associated with this hidden danger is adjusted accordingly according to the amplitude of the probability greater than its set upper threshold to improve the sensitivity of the sensor; when the occurrence probability of any type of hidden danger is less than its set lower threshold, the warning threshold of the sensor associated with this hidden danger is adjusted accordingly according to the amplitude of the probability less than its set lower threshold to reduce the sensitivity of the sensor;

[0114] When the input information resources related to the dam safety contain weather forecasts of the set type and degree, the warning threshold of the sensor associated with this weather forecast is adjusted according to the weather forecast to improve the sensitivity of the sensor;

[0115] Adjust the current warning threshold of the sensor according to the deviation between the historical warning threshold of the sensor and the historical hidden danger records;

[0116] Adjust the current warning threshold of the sensor according to the abnormal fluctuation of the sensor.

[0117] Specifically, the process of determining the emergency measure suggestions includes: when the occurrence probability of any type of hidden danger is greater than its set upper threshold, according to the technical standards and specifications contained in the input information resources related to the dam safety and the historical maintenance and reinforcement records of the corresponding hidden dangers in the engineering technical documents, the corresponding emergency measure suggestions for this hidden danger are obtained.

[0118] Embodiment 2: The method for constructing and operating a multi-modal large model for reservoir dam safety implemented by the above multi-modal large model construction and operation system for reservoir dam safety includes the following steps:

[0119] 1) The acquisition and processing of multi-modal sensor monitoring data inside the dam are realized by the real-time perception block of multi-modal dam internal monitoring data (Module 1). For example, Figure 2 As shown below, the specific technical details and implementation methods are as follows:

[0120] 1.1) Deployment and data acquisition of multi-modal sensors

[0121] The sensors mentioned above include physical quantity sensors, vibration sensors, visual sensors, audio sensors, and environmental sensors to obtain multi-modal data. Physical quantity sensors include deformation sensors, seepage sensors, stress-strain sensors, temperature sensors, etc.; visual sensors include cameras, drones, etc.; environmental sensors include meteorological sensors, seismic sensors, hydrological sensors, etc.

[0122] The real-time data obtained by the multi-modal sensors is transmitted to the central processing system through a wireless network and stored in a unified database for data processing, data fusion, and analysis of the multi-modal large model.

[0123] 1.2) Standardization and synchronous processing of multi-source heterogeneous data

[0124] (1) Standardization processing can eliminate the dimensional differences between various data and ensure that data of different modalities can be compared and fused in the same dimension. The formula for data standardization is:

[0125] ;

[0126] Among them, D i is the original data from the i th sensor, μ is the average value of the data of the i th sensor, is the standard deviation of the data of the i th sensor, D norm is the value of the data from the i th sensor after standardization processing. After standardization, all data is converted into standardized data with a mean of zero and a standard deviation of one, thus ensuring the relative consistency between data of different sensors.

[0127] (2) Timestamp alignment. The spatio-temporal synchronization algorithm is used to align the timestamps of data from different sensors to ensure that all modal data can be compared and analyzed in the same time dimension. The formula for time synchronization is:

[0128] ;

[0129] Among them, T sync is the synchronized timestamp,T i is the timestamp of each sensor, n and is the number of sensors. This algorithm ensures the consistency of data in the time dimension, especially in terms of time synchronization between vibration, audio, and visual data.

[0130] 1.3) Data Fusion and Error Handling

[0131] The data fusion of various sensors adopts a multi-modal weighted fusion method, and the formula for data fusion is:

[0132] ;

[0133] where, D i is the data from the i -th sensor, w i is the weight of the data of the i -th sensor. The system optimizes the fused data by adjusting the weight w i to make it more accurately reflect the actual state of the dam.

[0134] Considering the errors that may be introduced by the measurement accuracy and transmission delay of different sensors. Therefore, an error calculation formula is used to evaluate the accuracy of data fusion E f :

[0135] ;

[0136] This formula is used to calculate the error value generated during the fusion of different modal data. If the data of a certain sensor generates a large error after being fused with the data of other sensors, this module will dynamically adjust the data weight of this sensor w i to reduce the error and improve the accuracy of overall monitoring.

[0137] 2) Obtain external resource data related to dam safety monitoring and assessment based on network search, which is implemented by the external resource data real-time perception module based on network search (Module 2), including the acquisition and processing of external resource data based on network search, which is realized through automated web crawler technology. As Figure 3 shown, the acquisition and processing process of external resource data based on network search, and the specific technical details and implementation methods are as follows:

[0138] 2.1) Network data acquisition, including the following main steps:

[0139] (1) Index establishment and retrieval: An inverted index structure is established, and a B+ tree index structure is adopted. The retrieved data is sorted according to factors such as relevance and timeliness to ensure that users obtain the most useful information. The formula for the sorting score is:

[0140]

[0141] where CTR is the click-through rate, Relevance is the information P and the query Q relevance, Freshness is the timestamp of information release, and α, β, and γ are the CTR, Relevance, Freshness corresponding weights respectively. The click-through rate (CTR) is obtained by recording the display times and click times of each retrieval result. When a user conducts a search, the system automatically counts these data to calculate the CTR value. Relevance can be determined through manual annotation, user feedback, and machine learning models. Experts evaluate the relevance of each document and adjust it based on the user's click behavior and satisfaction feedback. In addition, the machine learning model can automatically score by analyzing features. The timeliness is obtained by relying on the recorded timestamp, and the system evaluates the freshness of the information based on the difference between the current time and the record creation time. Through these methods, the system can comprehensively judge the quality and relevance of each retrieval result. Through this ranking algorithm, this module can quickly obtain and utilize the latest information related to dam safety.

[0142] (2) Data caching and updating: A caching mechanism is adopted, and the buffered data is updated regularly. To improve the retrieval efficiency, a caching mechanism is adopted. Frequently accessed data, such as specific standards and regulations, is cached in local storage to reduce repeated queries; the cached data is updated regularly to ensure data timeliness.

[0143] (3) Parallel query and distributed computing: A distributed index scheme is adopted, and external data is distributed among multiple servers or nodes for processing. During querying, tasks are assigned to multiple servers for parallel processing, thereby reducing the load pressure on a single server and reducing the query response time, which can further improve the query efficiency.

[0144] 2.2) Network data fusion and application. By obtaining network data in real time, such as global patent information, international standards, laws and regulations, and the latest academic papers, the system can combine these external data with internal sensor data. First, the external data provides the latest technical and regulatory references for dam safety assessment. Second, the sensor data provides the real-time monitored health status of the dam. The system integrates these two types of data, analyzes potential risks and hidden dangers, and generates a comprehensive assessment report to support intelligent decision-making and ensure the safe operation and management of the dam. For example: By obtaining global patent data in real time, the system can identify the latest monitoring and reinforcement technologies to help managers select appropriate technical solutions for dam maintenance. By retrieving international standards and laws and regulations, the system can help dam operators adjust their operation processes to ensure compliance and reduce potential risks caused by non-compliance. The latest academic papers and research results provide rich background information for managers to assist in scientific risk assessment and decision-making.

[0145] 3) Obtain the historical data of the dam, including constructing a knowledge base of the dam's basic data and performing efficient retrieval through a vector database, which is achieved by Module 3. As Figure 4 shown, the specific technical details and implementation methods are as follows:

[0146] 3.1) Construct a knowledge base of the dam's basic data. The knowledge base of the dam's basic data includes the dam's survey, design, and construction data, the dam's daily operation data, operation log records, fault log records, hidden danger records, operation and maintenance records of the water turbine units, operation and maintenance data of the hydraulic gates, dam repair and reinforcement records, etc. These documents cover detailed information in all aspects of dam construction, operation, maintenance, and repair, and constitute an important basis for the system to conduct analysis and decision support.

[0147] 3.2) Text chunking. Chunk the documents in the knowledge base of the dam's basic data to obtain text chunks. Text chunks represent different chapters, paragraphs, or sentences of the document. The purpose of chunking is to divide longer documents into smaller segments to facilitate subsequent processing and calculation, and improve processing speed and data operability while maintaining semantic integrity. Each text chunk contains meaningful semantic units to ensure that high-quality and accurate content can be obtained in subsequent information extraction steps.

[0148] 3.3) Text embedding. Use a text embedding model to process the text chunks, convert the text chunks into digital vectors to obtain text chunk vectors, and all the text chunk vectors form a vector database. The semantic information of the text is represented by vectors, and the finally obtained vector database is the knowledge base of the dam's basic data.

[0149] Text embedding is achieved through machine learning algorithms. Text embedding models include Word2Vec, GloVe, and the deep learning-based BERT model, etc. The model learns the context relationships in a large corpus and maps similar words or sentences into a similar vector space. The closer two text blocks are semantically, the closer their distances are in the vector space. Through text embedding, the system can capture the subtle differences and hidden semantic information in the text, providing a basis for information extraction and relevance analysis.

[0150] 3.4) Information extraction and retrieval are based on the vector database.

[0151] Information extraction and retrieval are realized through the nearest neighbor search algorithm, which can quickly find the text block vector closest to the query vector and return the corresponding original text content. The nearest neighbor search algorithm includes the K-nearest neighbor algorithm or the approximate nearest neighbor search algorithm method 。

[0152] In practical applications, this function can quickly find records or knowledge related to the current problem from a large number of documents. For example, if the system needs to check the maintenance records of a specific dam, it can extract the maintenance and repair data related to the dam from the dam basic information knowledge base for risk assessment or decision support.

[0153] 4) Based on the real-time monitoring data and the dam historical data, a dynamic assessment and analysis of the dam safety hazards are carried out to determine the type and location of the hazards, which is implemented by Module Four. As Figure 5 shown, the specific technical details and implementation methods of the dam safety hazard assessment process are as follows:

[0154] 4.1) Construction of a refined BIM (Building Information Modeling) model of the dam. The BIM model represents the geometric information, material properties, construction history, and maintenance records of the dam through three-dimensional visualization, providing accurate basic information for the analysis of mathematical models, physical models, and mathematical-physical hybrid models.

[0155] For example, the system can identify the vulnerable parts of the dam or the areas with maintenance records in the past through the BIM model. These areas are often the key points of assessment and may have potential safety hazards.

[0156] 4.2) Analysis of dam safety hazards by integrating mathematical models, physical models, and mathematical-physical hybrid models. By combining the calculation results of the BIM model and mathematical models, physical models, and mathematical-physical hybrid models, a comprehensive safety assessment of the dam is carried out, and potential hazard areas are accurately located to determine the type and location of the hazards.

[0157] Mathematical models, physical models, and mathematical - physical hybrid models include finite element models, fluid mechanics models, heat conduction models, and soil mechanics models.

[0158] The BIM model provides geometric information, material properties, historical records, and foundation data for the calculations of the BIM model and mathematical models, physical models, and mathematical - physical hybrid models. The three - dimensional geometric structure of the BIM model consists of multiple structural units. After the geometric information (including volume, shape, and position) of each structural unit is discretized, it is transferred to the finite element model, fluid mechanics model, and heat conduction model for analysis to improve the accuracy of simulation calculations. Material properties include the strength, elastic modulus, heat conduction coefficient, etc. of concrete and are used for the mechanical analysis of the dam. Historical records include the historical maintenance records and stress history information of the dam and are used to identify areas where stress concentration has occurred or parts that have been repaired, so as to better judge whether there are repeated potential hazards in these areas.

[0159] The potential hazards of the dam include, but are not limited to, stress concentration, water flow scouring, cracks caused by thermal stress, and foundation instability, etc.

[0160] Stress concentration hazard: The finite element analysis model is used to simulate the stress and deformation distribution of the dam under different load conditions. The geometric information and material property data provided by the BIM model are discretized into multiple structural units, and the stress changes of these units under static loads (such as water pressure) and dynamic loads (such as earthquakes or floods) are calculated. Through stress tensor and deformation calculations, stress - concentrated areas are identified. These areas are often the weak links of the structure and may cause cracks, deformations, or even failures. For example, when a certain part of the dam bears excessive pressure or stress, the finite element analysis model can identify this position and, combined with the historical records of the BIM model, judge whether it is outside the safety threshold.

[0161] Water flow scouring hazard: The fluid mechanics model is combined with the BIM model to analyze the dynamic impact of water flow on the dam. Based on the dam body contour and boundary conditions provided in the BIM model, the fluid mechanics model simulates the interaction between water flow and the dam structure and predicts areas where structural damage may occur due to excessive water pressure or water flow scouring. The structural geometric information provided by the BIM model provides accurate boundary conditions for fluid mechanics analysis, enabling the fluid mechanics model to simulate real water flow behavior and thus more accurately identify water - pressure - sensitive areas.

[0162] Temperature change hazards: The heat conduction model simulates the thermal stress distribution of the dam under different temperature conditions, combines the material properties and structural locations provided by the BIM model, and identifies the expansion or contraction of materials caused by temperature changes. Under extreme temperature conditions (such as extremely cold or hot climates), the dam materials will generate thermal stress, leading to cracks or deformations. By simulating the stress distribution caused by temperature changes, the system can identify the affected parts and judge their stability.

[0163] Foundation instability hazards: The soil mechanics model evaluates the stability of the dam foundation by analyzing the interaction between the dam and the foundation soil. The foundation data provided by the BIM model is combined with the soil mechanics model to evaluate the bearing capacity and sliding risk of the foundation. By analyzing the possible structural instability caused by foundation settlement or slip, potential hazards of the dam foundation are identified, especially the areas prone to settlement, landslide or collapse, thus ensuring the safety of the dam foundation. The common soil model is the Mohr-Coulomb model, which can evaluate the supporting effect of the soil on the dam and analyze the stability of the soil under extreme conditions.

[0164] 4.3) Hazard marking: Mark the hazards analyzed by the above models on the BIM model and determine the specific locations of the hazards in the dam structure.

[0165] The BIM model not only provides geometric and material information, but also helps to accurately locate the hazards through its three-dimensional visualization function.

[0166] Hazard marking: When potential hazards are identified by mathematical, physical models and their hybrid models, mark these hazards on the corresponding structural units of the BIM model. For example, when the finite element analysis identifies a stress concentration area, highlight this area in color on the BIM model, which can be used to quickly identify the location of the hazard.

[0167] Hazard location positioning: Determine the specific location of the hazard in the dam structure through the three-dimensional BIM model. The three-dimensional BIM model can view different areas of the dam from multiple perspectives and understand the spatial distribution of the hazards. Ensure the accurate positioning of the hazard location, which is convenient for on-site maintenance and treatment.

[0168] 5) Based on the historical data and real-time monitoring data of the dam, identify the associated hazards to trace the propagation path of the hazards in the dam, which is realized by Module Five. As Figure 6 shown, the analysis process of the dam hazard propagation path, the specific technical details and implementation methods are as follows:

[0169] 5.1) Construct the dam knowledge graph and graph database.

[0170] Abstract the structure of the dam as a graph G=(V, E), where, Vis a set of nodes, representing each structural unit of the dam. The structural units include the dam body, spillway, foundation, etc.; E is a set of edges, representing the connection relationships between structural units. Among them, each node contains the attribute information of the corresponding structure. The attribute information includes historical detection data, material strength, stress distribution, etc. Each node v 's attributes are represented as a vector: A ( v ) = [historical detection data, material strength, stress distribution]. Each edge contains the physical connection of the structure and related attribute information. Each node v and node j The connecting edge e vj 's attributes are represented as: W ( e vj ) = [material strength, force transmission relationship]. The hidden danger propagation relationship is a dynamic attribute, which is closely related to the state of the node (such as the probability of occurrence of hidden dangers) and the propagation weight of the edge (such as the probability of hidden danger influence, the intensity of hidden danger propagation). This type of information usually needs to be dynamically generated through real-time calculation or historical data analysis, rather than the static attributes of fixed edges. Therefore, only the static attributes (material strength and force transmission relationship) are listed in the initial definition. In this embodiment, the main purpose of constructing edge attributes is to describe the physical and mechanical relationships between structural units to support basic structural analysis (such as stress transmission). The complexity of the hidden danger propagation relationship is relatively high and may be divided into a separate calculation module, rather than the default static attribute of the edge.

[0171] Store the set of nodes and the set of edges in a graph database, efficiently manage the structural data through the graph database, and provide basic materials for subsequent hidden danger propagation analysis. This structured data storage method makes data query and analysis more efficient and can provide accurate information in a short time. The graph database contains a set of nodes, a set of edges, and also contains the node connectivity used to represent the degree of association between a certain structural unit and other structural units.

[0172] Node connectivity LJ ( v ) is used to measure the relative importance of node v in the dam structure, specifically reflecting the connection degree between this node and other structural units. By calculating the connectivity of node v , its position and role in the overall dam structure can be judged. If the connectivity of a certain node is relatively high, it means that it is connected to multiple other nodes, indicating that this node may play an important function or bear greater stress in the structure. Node v Connectivity LJ ( v ) can be expressed as:

[0173] ;

[0174] Among them, A jv is an element in the adjacency matrix, representing whether node j is connected to node v , n is the total number of nodes in the knowledge graph.

[0175] The adjacency matrix is a matrix used to represent the graph structure, which can clearly show the connection relationship between nodes. If a certain node has a high degree of connection, it means that the node is in a key position in the dam structure and may have a greater impact on the overall safety of the dam.

[0176] The Pagerank algorithm is used to evaluate the relative importance of each node in the dam. The calculation formula is:

[0177] ;

[0178] Among them, PR ( v ) is the relative importance score of node v , B v is the set of nodes connected to node v , L ( u ) is the out-degree of node u , d is the damping factor (usually taking a value around 0.85), n is the total number of nodes. This score can help the system identify the key nodes in the dam structure. Especially when the dam is under stress or has potential hazards, these key nodes may be the parts that need the most attention.

[0179] 5.2) Identification of associated potential hazards and analysis of potential hazard propagation. Near the potential hazard location determined in step 4), identify possible associated potential hazards. The core of the knowledge graph is indeed composed of nodes and edges. Nodes represent entities (such as the structural units of the dam, potential hazards, etc.), and edges represent the relationships between nodes (such as connection, influence, etc.). However, the richness and functionality of the knowledge graph are not only reflected in the connection of nodes and edges, but also include the attribute information related to nodes and edges.

[0180] Node attributes: Each node can carry rich attribute information, such as historical detection data, material strength, stress distribution, etc. These attributes provide important background for understanding the characteristics and states of nodes. For example, the material strength and historical detection data of a certain structural unit of the dam can help evaluate the safety of the unit.

[0181] Edge Attributes: The attributes of edges describe the relationship characteristics between nodes, such as force transmission relationships or connection strengths. These attributes help analyze the interactions between different structural units. Especially when potential hazards occur, it is crucial to understand how forces are transmitted within the structure.

[0182] Information Integration: A knowledge graph can integrate information from different sources, such as historical hazard records, maintenance logs, and expert advice. This information can supplement nodes and edges, helping to construct a more comprehensive hazard network. For example, the location of a certain hazard may be related to the specific state of a node in historical records, and this relationship can be described by an edge in the knowledge graph.

[0183] Hazard Propagation Analysis: By analyzing the characteristics of nodes, their attributes, and edges, a knowledge graph can identify the associations between hazards and trace the propagation paths of hazards within the dam. This propagation analysis not only depends on the connectivity of nodes and edges but also comprehensively considers attribute information to more accurately assess the impact of hazards on the entire structure.

[0184] Therefore, the value of a knowledge graph lies in its ability to provide a profound understanding and analysis of complex structures through rich node and edge attribute information, thus effectively supporting hazard identification and propagation analysis.

[0185] To analyze hazard propagation, the system constructs a hazard impact matrix. The hazard impact matrix is a structured data table used to represent the hazard associations between different structural units of the dam. In this matrix, rows and columns represent each node, and the value of each cell reflects the intensity or risk level of hazard propagation between two nodes. These values can be calculated based on factors such as historical hazard records, the material strength of nodes, and stress distributions. By analyzing the hazard impact matrix, managers can identify high-risk nodes and accordingly formulate timely preventive and repair measures, thereby effectively reducing potential risks and ensuring the safe operation of the dam. By analyzing this matrix, managers can identify high-risk nodes and take timely measures.

[0186] Based on the graph database, the shortest path calculation is performed to analyze the propagation path of hazards in the dam structure. The formula for the shortest path is:

[0187] ;

[0188] where d ( u , v ) represents the shortest path distance between node u and node v , w is the weight of edge e i on the path, kis the number of edges on the path. In dam monitoring, the weight setting of edges is determined based on multiple key factors. First of all, physical distance is an important consideration. The shorter the distance between the connected nodes, the lower the weight of the edge is usually set, because the possibility of hidden danger spreading within a short distance is higher. In addition, the intensity of force transmission also affects the weight. The higher the strength and stiffness of the connecting material, the greater the resistance to the spread of hidden danger, and the corresponding weight can be lower. Historical hidden danger data is also important. If hidden danger spread often occurred on a certain path in the past, the weight of the edge should be adjusted lower to reflect a higher risk. Environmental factors such as rainfall, temperature changes, and material properties also affect the weight setting. By comprehensively considering these factors, a reasonable weight can be set for each edge, so that the shortest path calculation is more accurate, ensuring the scientificity and effectiveness of hidden danger spread analysis. The possibility and speed of hidden danger spreading from one node to other nodes are calculated through the shortest path algorithm.

[0189] In addition, the graph database also supports complex path traversal queries, which can efficiently traverse relevant paths in the dam structure according to the conditions set by users and provide detailed information on the paths (such as the status of each node and the attributes of edges). The path traversal query methods include using recursive traversal algorithms to query nodes v and nodes u all paths between P ( u , v ) and evaluate the importance of the path according to the edge weight w and node status on the path. In hidden danger spread analysis, the node status refers to the health or safety status of each node at a specific moment, reflecting its current security and potential risks. The node status usually includes the security level, such as "safe", "warning" or "dangerous", which is evaluated through real-time monitoring data (such as stress, displacement and temperature). In addition, if a certain node has been identified as having hidden dangers, its status will include the type of hidden danger, the severity and the possible scope of influence. Historical monitoring records and maintenance logs also affect the node status, especially the nodes where hidden dangers have occurred in the past need more attention. Environmental factors, such as rainfall or earthquake, will also affect the node status, changing its security. By comprehensively evaluating the node status, the system can effectively predict the risk of hidden danger spread and provide an important basis for management decision-making

[0190] If a hidden danger occurs at a certain node, the system will calculate the impact of the hidden danger on its adjacent nodes and predict the possible scope affected by the hidden danger through the spread model. The hidden danger spread probability can be expressed by the following formula:

[0191] ;

[0192] where F ( u , v ) is the probability of the hidden danger spreading from nodeu The probability of propagation to the node v , P ( u ) is the probability of potential hazards occurring at the node u . d ( u , v ) is the shortest path distance between the node u and the node v . This formula indicates that the likelihood of potential hazard propagation decays as the distance between nodes increases. Based on the propagation prediction results, the system can trigger an early warning mechanism. Especially when critical nodes are threatened, the system will automatically issue a warning to prompt the dam managers to conduct further inspections or take preventive measures. d 0 is the influence radius set by the system.

[0193] The probability of potential hazards occurring at the node P ( u ) can be achieved through various methods. First, historical data analysis is an effective approach. The probability is estimated by calculating the ratio of the frequency of potential hazards occurring at this node in the past to the total number of monitoring times. In addition, real-time monitoring data can also provide important information. By using data such as stress, displacement, and temperature monitored by sensors and combining with the set safety thresholds, the health status of the node is evaluated to judge the likelihood of potential hazards occurring. Constructing a statistical model is also a common method, which predicts the probability of potential hazards through various influencing factors (such as material properties, environmental conditions, etc.). Finally, expert evaluation can provide a basis for the probability, and the characteristics of the node are analyzed in depth by combining professional knowledge. Through these methods, the system can dynamically update the probability of potential hazards occurring at the node, providing accurate support for potential hazard propagation prediction.

[0194] 5.3) The dam updates the knowledge graph and the graph database. The data of the nodes and edges in the knowledge graph and the graph database are continuously incrementally updated as the monitoring data is updated, which is implemented by the incremental update algorithm.

[0195] Update the status information of the node according to the real-time sensor data, including the structural health status, stress conditions, etc. At the same time, the attributes of the edges are also dynamically adjusted. For example, when abnormal force transmission is found in certain areas during the structural inspection, the weights of the corresponding edges are adjusted, and the shortest path and node importance are recalculated.

[0196] By introducing the incremental update algorithm, only the changed parts of the nodes and edges are updated, greatly improving the response speed and operation efficiency of the system.

[0197] 6) The pre-training and fine-tuning of the multi-modal large model are implemented by Module 6. The multi-modal large model is pre-trained and fine-tuned to learn the risk assessment results, warning thresholds, and warning response mechanisms based on the real-time monitoring data, external resource data, dam historical materials, hidden danger locations and types, and dam hidden danger propagation paths. As Figure 7 shown, the technical details and implementation methods of the pre-training and fine-tuning of the multi-modal large model are as follows:

[0198] 6.1) Dataset construction and preprocessing.

[0199] Dataset construction: Collect historical multi-modal monitoring data (from Module 1), external resource data (from Module 2), dam historical materials (from Module 3), hidden danger locations and types (from Module 4), and hidden danger propagation paths (from Module 5). Based on the above data, input samples are formed: sensor data deployed on the dam, historical hidden danger records, engineering technical documents formed in each stage of the dam's planning, design, construction, and operation, information resources related to dam safety, geometric and material information of the BIM model, timestamp, and spatial location information.

[0200] Data preprocessing: Includes data cleaning (removing invalid data, noise, and outliers generated by sensor failures), format unification (such as equal-interval processing of time series data, and vectorization of text data through word segmentation and keyword extraction), alignment and synchronization (realizing time synchronization of multi-modal data based on timestamps, and aligning sensor data with the BIM model through spatial coordinates), and normalization processing (continuous data uses Min-Max Scaling, and categorical data is one-hot encoded).

[0201] Data labeling: Create labels for each sample, including binary labels indicating whether a hidden danger event has occurred (0: not occurred, 1: occurred), the probability value of the hidden danger event occurring (calculated by statistically calculating the ratio of the number of hidden danger occurrences under specific conditions to the total number of occurrences of this condition), and the type and impact degree of the hidden danger event, as auxiliary labels for the event classification task. Among them, the calculation data for the data labels comes from the dam historical materials (from Module 3), hidden danger locations and types (from Module 4), and hidden danger propagation paths (from Module 5).

[0202] Data augmentation: In the case of insufficient or unbalanced data, the generalization ability of the model can be improved through data augmentation techniques, including cutting long time series into short time windows to extract more features, jointly sampling multi-modal data to generate new samples, and using simulation methods to generate synthetic samples of sensor data or hidden danger propagation paths, so as to enrich the diversity and integrity of the dataset.

[0203] Modal Feature Encoding: The Transformer model requires serialized vectorized feature inputs. Therefore, feature encoding is performed on each modal data: Sensor data extracts time series features using a sliding window, visual data extracts image features through a pre-trained CNN (such as ResNet or ViT), text data generates embedding vectors using a pre-trained language model (such as BERT), BIM model data extracts geometric and material properties and numerically encodes them, and spatio-temporal information encodes timestamps and spatial positions into numerical vectors. These features are processed separately by independent encoders, providing a basis for the fusion of multi-modal features for the model.

[0204] Data Organization into Transformer Input Format: The Transformer architecture requires organizing multi-modal features in a sequential form. Each sample forms a sequence, where each element corresponds to a feature vector of a modality (e.g., Modality 1 Feature, Modality 2 Feature, Modality 3 Feature,... Modality 1 Feature, Modality 2 Feature, Modality 3 Feature,... Modality 1 Feature, Modality 2 Feature, Modality 3 Feature,...). The fusion of modal features is achieved through the multi-head self-attention mechanism of the Transformer, generating a contextually relevant joint representation, providing a basis for the comprehensive analysis of multi-modal data.

[0205] Saving the Training Set: To facilitate the efficient training and validation of the model, the training set needs to be saved in a format suitable for batch loading (such as.csv,.json, or.tfrecord), and the dataset is divided according to certain rules. Usually, the data is divided into a training set and a validation set by chronological order or random ratio. For example, 80% of the data is used for training and 20% for validation, thus ensuring the scientificity and accuracy of model training and evaluation.

[0206] 6.2) Model Fine-tuning and Pre-training

[0207] Based on the dataset established above, the multi-modal large model performs data fusion on the features of each modality through the multi-head self-attention mechanism of the Transformer. These data from different sources are aggregated into the Transformer model for joint analysis, enabling the model to provide information support from multiple dimensions when capturing potential hazards. Through the joint processing of multi-modal data, the system can comprehensively evaluate the health status of the dam by combining abnormal signals in the audio with surface change information in the image.

[0208] The loss function for the pre-training of the deep learning model based on the Transformer architecture is as follows:

[0209] ;

[0210] where, y iis the actual label, is the result predicted by the model, N is the total number of samples, and θ is the model parameter. This loss function is used to evaluate the difference between the model prediction result and the actual result.

[0211] To reduce the loss value of the model and optimize the model parameters, the gradient descent algorithm is introduced, and the formula is as follows:

[0212] ;

[0213] where θ is the learning rate, is the gradient of the loss function with respect to the parameter. By continuously updating the model parameter θ, the model is gradually optimized so that it can process and analyze the dam monitoring data more accurately.

[0214] To prevent the model from overfitting during training, the L2 regularization technique is introduced to maintain the generalization ability of the model, that is, even when facing unseen data, the model can make accurate predictions. The formula is as follows:

[0215] ;

[0216] where λ is the regularization parameter, which is used to control the complexity of the model and prevent the model from over-relying on the training data.

[0217] 6.3) Continuous learning and optimization of the model. Through the online learning mode, new monitoring data is used to continuously update the model to maintain its sensitivity to environmental changes.

[0218] The key to online learning is the rapid response to new data and the dynamic adjustment of the model weights. For example, during floods or extreme weather events, the latest monitoring data is used to quickly update the model parameters to improve the model's ability to respond to emergencies. At the same time, the closed-loop feedback mechanism of the system allows the comparison of each prediction result with the actual data, obtaining the error and adjusting the model to maintain stable high performance during long-term monitoring.

[0219] During the continuous learning and optimization process of the model, the error refers to the difference between the model's prediction result and the actual observed value. This difference can be quantified in various ways. For example, the prediction error is the deviation between the model's output for the input data and the true label. The loss function is used to comprehensively evaluate this error, such as commonly used loss functions like mean squared error and cross-entropy. In online learning, the system, through a closed-loop feedback mechanism, compares the model's prediction result with the latest monitoring data each time and calculates the feedback error. This process not only helps the model adjust its weights and parameters but also decomposes the error into bias and variance to further optimize the model's performance. By continuously monitoring and analyzing the error, the system can enhance its ability to handle emergencies in real time and ensure scientific and effective decision-making support for dam safety management.

[0220] Through continuous optimization, the model weights and parameters are continuously adjusted according to the latest data to ensure that it can always provide scientific and effective decision-making support for dam safety management in various complex environments.

[0221] 7) Comprehensive warning and response for dam safety based on multi-modal large models: Input real-time monitoring data, the type and location of potential hazards, the dam's historical data related to the potential hazards, the propagation path of the potential hazards within the dam, and external resource data related to the potential hazards into a pre-trained multi-modal large model. Analyze based on the dam's historical data and real-time external resource data to obtain the probability of potential hazards occurring in the dam, the degree of impact of the potential hazards, and the warning threshold, and then output the risk assessment result and warning response mechanism, which is achieved through Module 7.

[0222] As Figure 8 shown, for the comprehensive warning and response mechanism for dam safety based on multi-modal large models, the specific technical details and implementation methods are as follows:

[0223] 7.1) The multi-modal large model predicts the probability of potential hazards occurring in the dam by integrating real-time monitoring data, external resource data, and historical data P failure and the possible degree of impact brought by the potential hazards in the dam C impact , thereby generating the final risk assessment result.

[0224] (1) Probability of potential hazards occurring P failure : The multi-modal large model uses real-time monitoring data from multi-modal sensors, including information such as water flow pressure, vibration, temperature, and displacement, to analyze the current structural state of the dam.

[0225] These data reflect the operating conditions of the dam under different environmental conditions and provide rich feature vectors for the multi-modal large model. Through deep learning algorithms, the multi-modal large model can identify potential hazards and calculate their occurrence probabilities Pfailure In addition to the real-time monitoring data of internal sensors, the multi-modal large model also makes full use of external resource data and dam historical data to improve the accuracy of prediction. Through the comprehensive analysis of these multi-dimensional data, the multi-modal large model can more accurately judge the probability of potential hazards caused by dam maintenance records and other factors, and dynamically adjust P failure the value.

[0226] The external resource data is obtained through network search, providing a cutting-edge reference for the risk prediction of the multi-modal large model. For example, the analysis of accident cases worldwide can help the multi-modal large model identify potential hazard scenarios similar to the current situation of the dam, so as to adjust P failure more precisely.

[0227] The dam historical data comes from the dam basic information knowledge base constructed by RAG (Retrieval-Augmented Generation). By analyzing historical records, the multi-modal large model can capture the long-term operation trends, maintenance history and potential hazard distributions of the dam, and combine these data to predict the probability of current potential hazards. For example, if historical records show that potential hazards have occurred in certain areas multiple times, the multi-modal large model can increase the probability of potential hazards in these areas P failure the value.

[0228] (2) Degree of impact C impact : The multi-modal large model evaluates the potential consequences of risks by assessing the degree of impact that dam potential hazards may cause C impact . The degree of impact C impact not only depends on the probability of potential hazards, but also involves the threat level of the potential hazards to key parts of the dam. The multi-modal large model calculates the damage that each type of potential hazard may cause through the analysis of historical potential hazard records and dam maintenance data. C impact The calculation formula of

[0229] is:

[0230] Among them, I i represents the degree of impact of the i-th type of potential hazard event on the dam, is the probability of this type of potential hazard event, and n represents the number of potential hazard event types.. This formula integrates the risk levels of multiple potential hazards and assigns different weights to each potential hazard. The model obtains the comprehensive impact degree value based on factors such as the possible structural damage, economic losses, and ecological impacts brought by these potential hazards. By quantifying the impacts of various potential hazards, managers can make preparations for prevention and response in advance.

[0231] (3)Comprehensive Risk Score R： By combining the probability of potential hazards P failure with the degree of impact C impact to obtain the comprehensive risk score R :

[0232]

[0233] Among them, α p is the weight of the probability of potential hazards P failure ; β c is the weight of the degree of impact C impact . The weights α p and β c can be obtained through various methods. First, the weight values can be initially set with the help of expert experience and industry standards to ensure their rationality. Second, user feedback and the opinions of management personnel can be collected to understand the importance of the probability of potential hazards and the degree of impact on decision-making in actual operations, and then corresponding adjustments can be made. In addition, through A / B testing, different weight combinations can be used in different scenarios to compare their effects and find the optimal configuration. Using machine learning models is also an effective method. By training the model, the weights can be automatically optimized to better meet the actual needs. Finally, statistical analysis of historical data can help evaluate the changing trends of the probability of potential hazards and the degree of impact, providing data support for weight setting. Through these methods, the comprehensive risk score will be more accurate, providing an effective decision-making basis for the safety management of the dam.

[0234] The output of the multi-modal large model directly affects the final value of the comprehensive risk score R , thus providing data support for the risk management of the dam. In this way, management personnel can more comprehensively understand potential risks and formulate reasonable risk response strategies accordingly. For example, when R the value is relatively high, the system may recommend increasing the monitoring frequency, strengthening the maintenance work of the dam, or preparing emergency measures.

[0235] The multi-modal large model analyzes multi-modal data, calculates the probability of potential hazards and the degree of impact, providing accurate and intelligent support for the safety risk assessment of the dam. The comprehensive risk score generated by the system not only helps identify potential hazards but also provides a scientific basis for dam managers to ensure timely and reasonable risk response strategies.

[0236] 7.2) Triggering of Dam Safety Risk Early Warning. The multi-modal large model dynamically adjusts the early warning threshold by comprehensively analyzing internal sensor data, external resource data, and historical data. The specific adjustment mechanism is as follows:

[0237] (1) Threshold adjustment based on model output: When the multi-modal large model predicts that the occurrence probability P of a certain type of hidden danger failure exceeds the set threshold (e.g., 70%), the system will actively lower the alarm threshold of the sensors related to this hidden danger to improve the sensitivity of monitoring. The reduction amplitude is determined according to the degree to which the occurrence probability of the hidden danger exceeds the set threshold. For example, for every 5% that the occurrence probability of the hidden danger exceeds the threshold, the alarm threshold of the relevant sensors is reduced by 2% to 5%. Specifically, when the occurrence probability of the hidden danger is high, the threshold is lowered; when the occurrence probability of the hidden danger is low and the external environment is stable, the threshold can be moderately increased to reduce false alarms. For example, if the model predicts that the occurrence probability of the structural stress hidden danger in a certain area reaches 80%, exceeding the set threshold by 10%, the system will lower the alarm threshold of the stress sensors in this area by 5% to 10% to more sensitively capture stress changes.

[0238] (2) Threshold adjustment in response to external environment data: When the external environment forecast (such as heavy rain, earthquake, etc.) shows that it may affect the dam, the system dynamically adjusts the alarm threshold of the relevant sensors according to the degree of environmental impact. In the case of expected heavy rain, the system lowers the alarm thresholds of the seepage sensors and water level sensors, and the reduction amplitude can be adjusted according to the rainfall level (light rain, heavy rain, heavy rainstorm), for example, reduced by 5% to 15%. In the case of earthquake early warning, the system adjusts the alarm thresholds of the ground stress and vibration sensors according to the earthquake magnitude and the distance between the epicenter and the dam. The greater the magnitude and the closer the distance, the greater the reduction amplitude of the threshold, which may be reduced by 10% to 20%. By lowering the threshold, the sensitivity of the sensors to abnormalities caused by external environmental changes is improved, ensuring that potential risks can be detected in a timely manner.

[0239] (3) Threshold optimization based on historical data: The system regularly analyzes the deviation between the sensor data and the historical hidden danger records, and dynamically adjusts the early warning threshold. When the data of a certain sensor triggers an alarm multiple times when it is 5% to 10% higher than the set threshold, the system determines that the current threshold may be too sensitive or insensitive. According to the degree of deviation, the threshold is appropriately increased or decreased, and the adjustment amplitude is generally between 5% and 10%. For example, if historical data shows that the incidence of a certain type of hidden danger increases under specific environmental conditions (such as continuous high temperature), the system will lower the threshold of the relevant sensors during this period to improve the monitoring sensitivity.

[0240] (4)Threshold anomaly detection and adaptive optimization: The system dynamically adjusts the warning threshold using anomaly detection algorithms by monitoring the changes in sensor data in real time. When the sensor data shows abnormal fluctuations (such as drastic changes within a short period) and does not conform to the historical trend, the system identifies it as an anomaly. If the anomaly persists for more than a preset time (e.g., 30 minutes), the system combines historical data and external information to appropriately adjust the threshold, with an adjustment range of 5% to 10%. When data anomalies are caused by equipment aging or external interference, the threshold is adjusted to avoid false alarms or missed alarms.

[0241] (5)Comprehensive assessment and threshold adjustment: The system comprehensively considers the probability of potential hazards output by the model, external environmental changes, and historical data to determine the direction and amplitude of threshold adjustment. In high-risk scenarios, when the probability of potential hazards is high and the external environment is unfavorable, the threshold is lowered by up to 20%, and the monitoring frequency is increased. In low-risk scenarios, when the probability of potential hazards is low and the environment is stable, the threshold can be moderately increased by 5% to 10% to reduce false alarms and improve the system efficiency. The adjustment of the threshold is directly based on the probability of potential hazards output by the model. The higher the probability, the more the threshold is lowered, and the more sensitive the monitoring system becomes. This dynamic adjustment mechanism ensures the accuracy and flexibility of the early warning system.

[0242] 7.3) Dam safety risk early warning response mechanism. The dam safety risk early warning response mechanism is a key system that ensures managers can quickly take effective response measures when the dam faces potential risks. This mechanism relies on the output of the multi-modal large model, combines external resource data and dam historical data, and provides accurate and dynamic early warning response strategies for dam management. Through the network search function to obtain the latest global standards and specifications, academic papers, and laws and regulations, as well as the dam basic information knowledge base constructed based on the RAG technology, the system can establish a perfect early warning response mechanism to ensure the scientific nature and timeliness of decision-making.

[0243] The multi-modal large model ensures that dam monitoring and emergency strategies comply with international standards by analyzing the latest technical standards and specifications retrieved by Module 2. For example, updates to flood and earthquake emergency response and monitoring equipment installation requirements are promptly incorporated into the early warning mechanism to help managers adjust emergency measures according to these standards, ensuring their compliance and scientific nature. Academic research provides the latest results for dam risk assessment, structural reinforcement, etc., supporting managers to formulate more scientific protection plans and enhancing the effectiveness of emergency measures. In addition, the system can obtain the latest laws and regulations to ensure that dam managers comply with relevant laws during disaster early warning, evacuation, and other emergency responses, avoiding legal risks.

[0244] Based on historical maintenance and reinforcement records, the multi-modal large model identifies high-risk areas and prioritizes the monitoring of these vulnerable parts during emergencies. The hidden danger log and operation log help the system analyze past hidden dangers and their handling methods, thus providing response strategies for early warning responses. For example, if hidden dangers have occurred in certain areas due to similar environmental conditions, the multi-modal large model will propose corresponding emergency plans based on historical data, such as increasing the monitoring frequency or implementing preventive reinforcement, to ensure that managers can quickly make targeted responses and reduce potential losses.

[0245] By combining the external resource data obtained through the network search function with the dam historical data provided by the RAG knowledge base, the multi-modal large model finally forms a comprehensive risk early warning response mechanism. The system can provide accurate early warning signals when the dam is facing risks, and combine standards and specifications, academic research, laws and regulations, as well as historical maintenance and reinforcement data to provide comprehensive emergency decision-making support for managers.

[0246] This mechanism ensures that managers can not only formulate emergency plans based on the latest standards, specifications, laws and regulations, but also combine the historical operation data of the dam to identify high-risk areas and hidden danger propagation paths, and accurately formulate protection strategies. In the emergency state, the system can dynamically adjust the response plan according to the real-time analysis of multi-source data to ensure the scientificity, timeliness and compliance of emergency measures, and effectively guarantee the safety and reliability of the dam.

[0247] The content not described in detail in this specification belongs to the prior art well-known to those skilled in the art.

Claims

1. A multi-modal large model construction and operation method for reservoir dam safety, characterized by: The following steps are involved: Constructing training sets: Collect historical multimodal monitoring data, external resource data, dam historical data, hidden danger locations and types, and hidden danger propagation paths, and form input samples based on the above data; the calculation data of training labels comes from dam historical data, hidden danger locations and types, and hidden danger propagation paths; The process of obtaining the hidden danger propagation path includes: defining each structural unit of the dam as a structural unit node and assigning attributes; defining the physical connection and mechanical relationship between the structural units as a structural connection edge and assigning attributes; Establish the relationship between nodes and edges: Establish the edges between the nodes of the structural units based on the historical operation status data, engineering technical documents and geometric information in the BIM model; Store the node and edge information in the graph database; calculate the nodes and edges in the graph database to obtain the node connectivity and importance scores; Based on the real-time collected dam operation status data and new external information resources, the health status and risk level of the nodes in the knowledge graph, as well as the weight and impact probability of the edges, are updated; Using the updated node and edge attributes and the results of connectivity calculation, the path and probability of hidden dangers propagating from one structural unit to another are analyzed; Based on the probability of hidden danger propagation, a matrix representing the hidden danger correlation between nodes is constructed; combining the hidden danger impact matrix and the graph database, the shortest path from the source node to other nodes of the hidden danger is calculated; according to the importance score of the node and its role in the hidden danger propagation process, the nodes with important influence in the hidden danger propagation process are identified; A single sample of the training set includes: multimodal data as model input: sensor data deployed on the dam, historical hidden danger records, engineering and technical documents formed at the planning, design, construction and operation stages of the dam, information resources related to dam safety, geometry and material information of the BIM model, timestamp and spatial location information; and whether a hidden danger event occurs and the probability value of the hidden danger event occurring as training labels; Use the training set to train the multimodal pre-trained model based on the Transformer architecture; The real-time monitoring data, the type and location of hidden dangers, the historical data of the dam related to the hidden dangers, the propagation path of the hidden dangers in the dam, and the external resource data related to the hidden dangers are used to make real-time predictions and early warnings of dam safety hazards based on the trained model; Based on the probability value of whether the hidden danger event occurs output by the model, the actual impact degree of the hidden danger event, the comprehensive risk score, the warning threshold of the hidden danger event-related sensor and the emergency measure recommendations are calculated.

2. A method according to claim 1, characterized in that: After preprocessing, the data is used to construct the training set: the collected raw data of the training samples is cleaned to remove abnormal and invalid data; the data is annotated in a unified format and organized into structured data suitable for model training; The time alignment of image and vibration data is achieved through timestamps, and the spatial coordinates are uniformly processed.

3. A method according to claim 1, characterized in that: The sensors deployed on the dam include but are not limited to: physical quantity sensors, vibration sensors, visual sensor data, audio sensors and environmental sensors.

4. A method according to claim 1, characterized in that: During the model training process, the binary cross entropy loss function, gradient descent algorithm and L2 regularization are used to optimize the model parameters.

5. A method according to claim 1, characterized in that: Before training and using the Transformer-based multimodal pre-training model, the input data of each modality is preprocessed through an independent encoder, and the features of each modality are fused through the Transformer's multi-head self-attention mechanism and used as model input.

6. A method according to claim 1, characterized in that: The actual impact of hidden danger events is calculated using the following formula: impact : Among them, I i represents the impact of the i-th hidden danger event on the dam, is the probability of occurrence of this type of hidden danger event, and n represents the number of hidden danger event types.

7. A method according to claim 6, characterized in that: The comprehensive risk score R is calculated using the following formula: R=αP failure +βC impact Among them, α is the weight of the probability of hidden danger occurrence, indicating its relative importance in risk assessment, and β is the weight of the impact degree of hidden danger, indicating its contribution to the comprehensive score.

8. A method according to claim 3, characterized in that: The warning thresholds of the sensors associated with potential hazards are adjusted comprehensively according to the following principles and their corresponding weights: When the probability of occurrence of any type of hidden danger is greater than its set upper threshold, the warning threshold of the sensor associated with the hidden danger is adjusted accordingly according to the extent by which the probability is greater than its set upper threshold, so as to improve the sensitivity of the sensor; when the probability of occurrence of any type of hidden danger is less than its set lower threshold, the warning threshold of the sensor associated with the hidden danger is adjusted accordingly according to the extent by which the probability is less than its set lower threshold, so as to reduce the sensitivity of the sensor; When the input dam safety-related information resource includes a weather forecast of a set type and degree, adjusting the warning threshold of a sensor associated with the weather forecast according to the weather forecast to improve the sensitivity of the sensor; Adjust the current warning threshold of the sensor according to the deviation between the historical warning threshold of the sensor and the historical hidden danger record; Adjust the current warning threshold of the sensor according to the abnormal fluctuation of the sensor.

9. A method according to claim 3, characterized in that: The process of determining emergency measures recommendations includes: when the probability of occurrence of any type of hidden danger is greater than its set upper limit threshold, the corresponding emergency measures recommendations for the hidden danger are obtained based on the technical standards and specifications contained in the input dam safety-related information resources and the historical maintenance and reinforcement records of the corresponding hidden dangers in the engineering technical documents.

10. A multi-modal large model construction and operation system for reservoir dam safety, characterized by: Including training set construction module, model training module, prediction module and early warning module: The training set construction module is used to construct a training set: a single sample of the training set includes: multimodal data as model input: sensor data deployed on the dam, historical hidden danger records, engineering and technical documents formed at the planning, design, construction and operation stages of the dam, information resources related to dam safety, geometry and material information of the BIM model, timestamp and spatial location information; And whether the hidden danger event as the training label occurs and the probability value of the hidden danger event; The model training module is used to train a multimodal pre-trained model based on the Transformer architecture using a training set; The prediction module is used to make real-time predictions and early warnings of dam safety hazards based on the trained model using real-time collected data; The early warning module is used to calculate the actual impact of the hidden danger event, the comprehensive risk score, the early warning threshold of the hidden danger event-related sensors, and emergency measures based on the probability value of whether the hidden danger event occurs output by the model.

Images