Multi-modal large model construction and operation method and system for reservoir dam safety

Abstract

The invention provides a multi-modal large model construction and operation method and system for reservoir dam safety. The method comprises the following steps: constructing a training set, wherein the training set comprises a single sample; multi-modal data as model input: sensor data arranged on the dam, historical hidden danger records, engineering technology documents formed in various stages of planning, design, construction and operation of the dam, information resources related to dam safety, geometric and material information of the BIM model, timestamps and spatial position information; the probability value of occurrence of the hidden danger event serving as the training label is obtained; the training set is adopted to train a multi-mode pre-training model based on a Transform framework; performing real-time prediction and early warning on potential safety hazards of the dam; and calculating the actual influence degree, the comprehensive risk score, the dynamically set early warning threshold and the emergency measure suggestion of the hidden danger event according to the probability value of whether the hidden danger event occurs or not output by the model. The scientificity and effectiveness of dam safety monitoring are comprehensively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of dam safety protection, and in particular relates to a method and system for constructing and operating a multi-modal large model for reservoir dam safety. Background Art

[0002] As an important part of water conservancy projects, the safety of dams directly affects the safety of life and property of residents in the downstream area and the stability of the ecological environment. Therefore, the health monitoring and safety assessment of dams have always been an important topic in the global engineering and scientific research fields. With the increase in the service life of dams, the frequent occurrence of natural disasters and the extreme environment brought about by climate change, the demand for health monitoring of dam structures has become more urgent. How to conduct safety monitoring in real time and efficiently in a complex and changing environment has become a key challenge.

[0003] Traditional dam safety monitoring mainly relies on manual inspections and single-modal sensors, which have obvious deficiencies in real-time, accuracy, and comprehensiveness. Manual inspections are inefficient, rely on experience, and are difficult to cover all hidden dangers. Single-modal monitoring, such as strain sensors, can only obtain data on a specific aspect of the dam and cannot provide an overall health assessment. For example, strain sensors can detect anomalies but cannot identify problems such as surface cracks. The fusion and analysis of multimodal data is an important way to improve the comprehensiveness of monitoring, but the data of different sensor types are highly heterogeneous, and fusion analysis is difficult, and often cannot effectively reveal comprehensive risks.

[0004] Existing dam monitoring systems also have limitations in intelligent analysis and decision-making capabilities. Most systems rely on fixed thresholds to trigger warnings, lack flexibility, and cannot dynamically respond to complex environmental changes. In addition, existing early warning systems are usually unable to deeply analyze the causes of hidden dangers or provide effective response suggestions, making it difficult for managers to make scientific decisions in a timely manner.

[0005] The lack of historical data and knowledge base further limits the effectiveness of the existing system. The existing system mostly relies on real-time data, lacks correlation analysis with historical hidden danger records and maintenance and reinforcement data, and cannot make full use of past experience for intelligent decision-making. Historical information in dam operation is often of great significance for predicting potential risks, and the isolated analysis method of the existing system easily ignores the value of this information.

[0006] Insufficient real-time monitoring and dynamic prediction capabilities are also the shortcomings of current dam monitoring technology. In the face of climate change and natural disasters, the state of the dam will evolve dynamically over time, and the existing static monitoring methods are difficult to identify structural hazards in a timely manner and make effective warnings. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the above-mentioned background technology and to provide a method and system for constructing and operating a multimodal large model for reservoir dam safety, so as to comprehensively improve the scientificity and effectiveness of dam safety monitoring.

[0008] The technical solution adopted by the present invention is: a multi-modal large model construction and operation method for reservoir dam safety, comprising the following steps: Constructing a training set: A single sample of the training set includes: multimodal data as model input: sensor data deployed on the dam, historical hidden danger records, engineering and technical documents formed at the planning, design, construction and operation stages of the dam, information resources related to dam safety, geometry and material information of the BIM model, timestamp and spatial location information; and whether a hidden danger event occurs and the probability value of the hidden danger event occurring as training labels; Use the training set to train the multimodal pre-trained model based on the Transformer architecture; Use real-time collected data to conduct real-time prediction and early warning of dam safety hazards based on trained models; Based on the probability value of whether the hidden danger event occurs output by the model, the actual impact degree of the hidden danger event, the comprehensive risk score, the warning threshold of the hidden danger event-related sensors and the emergency measures recommended are calculated.

[0009] In the above technical solution, the data is preprocessed and then used to construct a training set: the collected original data of the training samples is cleaned to remove abnormal data and invalid data; the data is labeled in a unified format and organized into structured data suitable for model training; the time alignment of the image and vibration data is achieved through timestamps and the spatial coordinates are uniformly processed.

[0010] In the above technical solution, the sensors deployed on the dam include but are not limited to: physical quantity sensors, vibration sensors, visual sensor data, audio sensors and environmental sensors.

[0011] In the above technical solution, during the model training process, a binary cross entropy loss function, a gradient descent algorithm and L2 regularization are used to optimize the model parameters.

[0012] In the above technical solution, before training and using the multimodal pre-trained model based on the Transformer architecture, the input data of each modality is preprocessed through an independent encoder, and the features of each modality are fused through the Transformer's multi-head self-attention mechanism and used as the model input.

[0013] In the above technical solution, the following formula is used to calculate the actual impact degree C of the hidden danger event: impact : ; Among them, I i represents the impact of the i-th hidden danger event on the dam, is the probability of occurrence of this type of hidden danger event, and n represents the number of hidden danger event types.

[0014] In the above technical solution, the comprehensive risk score R is calculated using the following formula:

[0015] Among them, α is the probability of hidden danger occurrence P failure The weight of , indicating its relative importance in risk assessment, and β is the weight of the impact of hidden dangers, indicating its contribution to the comprehensive score.

[0016] In the above technical solution, the warning threshold of the hidden danger event associated sensor is comprehensively adjusted according to the following principles and their corresponding weights: When the probability of occurrence of any type of hidden danger is greater than its set upper threshold, the warning threshold of the sensor associated with the hidden danger is adjusted accordingly according to the extent by which the probability is greater than its set upper threshold, so as to improve the sensitivity of the sensor; when the probability of occurrence of any type of hidden danger is less than its set lower threshold, the warning threshold of the sensor associated with the hidden danger is adjusted accordingly according to the extent by which the probability is less than its set lower threshold, so as to reduce the sensitivity of the sensor; When the input dam safety-related information resource includes a weather forecast of a set type and degree, adjusting the warning threshold of a sensor associated with the weather forecast according to the weather forecast to improve the sensitivity of the sensor; Adjust the current warning threshold of the sensor according to the deviation between the historical warning threshold of the sensor and the historical hidden danger record; Adjust the current warning threshold of the sensor according to the abnormal fluctuation of the sensor.

[0017] In the above technical scheme, the process of determining the emergency measures recommendation includes: when the probability of occurrence of any type of hidden danger is greater than its set upper limit threshold, the corresponding emergency measures recommendation for the hidden danger is obtained based on the technical standards and specifications contained in the input dam safety-related information resources and the historical maintenance and reinforcement records of the corresponding hidden danger in the engineering technical documents.

[0018] The present invention also provides a multi-modal large model construction and operation system for reservoir dam safety, including a training set construction module, a model training module, a prediction module and an early warning module: The training set construction module is used to construct a training set: a single sample of the training set includes: multimodal data as model input: sensor data deployed on the dam, historical hidden danger records, engineering and technical documents formed by the dam in the planning, design, construction and operation stages, information resources related to dam safety, geometry and material information of the BIM model, timestamp and spatial location information; and whether a hidden danger event occurs and the probability value of the hidden danger event as a training label; The model training module is used to train a multimodal pre-trained model based on the Transformer architecture using a training set; The prediction module is used to make real-time predictions and early warnings of dam safety hazards based on the trained model using real-time collected data; The early warning module is used to calculate the actual impact of the hidden danger event, the comprehensive risk score, the early warning threshold of the hidden danger event-related sensors, and emergency measures based on the probability value of whether the hidden danger event occurs output by the model.

[0019] The beneficial effects of the present invention are as follows: the present invention integrates multimodal data sources (operation status data, historical records, technical documents, BIM models, etc.) to achieve comprehensive perception of the status of the dam throughout its life cycle. The breadth and depth of the data ensure that the model can accurately predict hidden danger events; through a multimodal pre-training model based on the Transformer architecture, a multi-head self-attention mechanism is used to efficiently capture the correlation between each mode, thereby improving the accuracy of hidden danger prediction; real-time prediction and early warning capabilities ensure timely response to potential safety hazards of the dam, providing reliable technical support for dam operation and management; providing actual impact level, comprehensive risk score and dynamic early warning threshold, providing a scientific basis for emergency response.

[0020] Furthermore, the present invention uses data cleaning and anomaly detection to remove invalid or interfering information, ensuring the high quality of training data and the robustness of the model; unified formatting and time alignment processing improves the structured degree of data and provides stable input for model training; the alignment of time and space information ensures the spatiotemporal consistency between multimodal data and enhances the model's ability to capture complex hidden danger patterns.

[0021] Furthermore, the multimodal data source of the present invention covers the key monitoring data (physical quantity, vibration, vision, audio, environment) in the operation of the dam, and comprehensively reflects the real-time status of the dam; different types of sensor data jointly provide a multi-dimensional source of information, improving the reliability and comprehensiveness of the model in identifying hidden danger events; including data input from environmental sensors, it takes into account the impact of external environmental changes on dam safety, and enhances the adaptability of the system.

[0022] Furthermore, the present invention adopts a binary cross entropy loss function to adapt to the classification problem of hidden danger prediction and ensure the optimization of classification accuracy; uses a gradient descent algorithm and L2 regularization to improve the training efficiency of the model, avoid overfitting, and enhance the generalization ability of the model; through the combination of optimization algorithms, the stability and predictive ability of the model in complex data scenarios are ensured.

[0023] Furthermore, the independent encoding method of each modal data of the present invention retains the feature independence between modalities, ensuring that information will not be lost in the early stages; the multi-head self-attention mechanism of the Transformer architecture integrates multimodal features, effectively captures high-order interactions between modalities, and improves the accuracy of hidden danger prediction; the modular structure of the model is easy to expand and can adapt to new data modalities or changing system requirements in the future.

[0024] Furthermore, the present invention quantifies the actual impact of hidden danger events by calculating the impact degree of hidden danger events, providing a basis for subsequent comprehensive risk scoring; the quantitative impact degree calculation provides a clear analysis basis for safety management, which helps decision makers to formulate more scientific response strategies.

[0025] Furthermore, the comprehensive risk score of the present invention integrates the probability of hidden dangers and the degree of impact, providing a more comprehensive safety evaluation index; introducing weight factors, flexibly adjusting the weights of hidden danger probability and impact in risk assessment, to meet the needs of different scenarios; the comprehensive risk score provided lays the foundation for warning threshold setting and emergency measures recommendations.

[0026] Furthermore, when the probability of a hidden danger event exceeds the set upper threshold, the present invention automatically increases the sensitivity of the relevant sensor so that potential safety hazards can be detected earlier and more accurately; when the probability of a hidden danger event is lower than the lower threshold, the sensitivity of the sensor is correspondingly reduced, thereby avoiding overreaction of the sensor and reducing unnecessary warnings. When the sensor is associated with weather forecast data, the present invention can adjust the warning threshold of the sensor according to weather changes, especially in extreme weather conditions, to give early warnings and enhance the risk resistance of the dam. Based on the deviation between the historical warning threshold of the sensor and the hidden danger record, and the abnormal fluctuation of the sensor, the present invention dynamically adjusts the current warning threshold to avoid misjudgment or missed judgment due to equipment failure or historical record deviation. By adjusting the sensitivity of the sensor in real time, the present invention can more accurately reflect the risk status of the actual hidden danger event, avoid false alarms and missed alarms, and can make comprehensive judgments based on various information, timely adjust the working status of the equipment, thereby improving the overall safety of the dam. Combined with multimodal data and external factors (such as weather forecasts, historical data, etc.), the early warning system can be adaptively adjusted under changing environmental conditions to improve the emergency response capability of the dam.

[0027] Furthermore, the present invention proposes a process for determining emergency measures recommendations through technical standards, engineering technical documents and historical maintenance records. When the probability of a hidden danger event exceeds a set threshold, the system will provide standardized emergency measures recommendations for specific hidden danger types based on the design standards, construction specifications and operation requirements of the dam. This ensures that the emergency measures meet industry standards and ensure their effectiveness and feasibility. Based on the maintenance and reinforcement records of historical hidden dangers, the system can extract past experience in solving similar problems and provide specific solutions for current hidden dangers. The application of historical experience helps to avoid repetitive errors and improve the efficiency and quality of emergency response. The present invention can automatically generate emergency measures based on the probability of occurrence of real-time hidden danger events and related data, reduce manual intervention, and improve the speed of emergency response; through the combination of historical maintenance records and technical standards, the system can convert accumulated experience and technical specifications into emergency response plans, enhancing the system's intelligence level and decision-making support capabilities; timely and targeted emergency measures recommendations help improve the safety protection capabilities of the dam and reduce losses and risks after an accident.

[0028] Furthermore, the present invention can not only improve the intelligence level of the dam safety monitoring system, but also enhance its early warning and emergency response capabilities for hidden danger events. Through the dynamic adjustment of the sensor early warning threshold and the intelligent generation of emergency measures, the potential risks of the dam can be effectively predicted, evaluated and processed in a changing environment, thereby providing more reliable protection for the long-term safe operation of the dam. The application of these technical solutions can greatly reduce the probability of serious accidents in the dam, improve its risk management level, and ultimately ensure the safety and social benefits of the dam. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is the overall architecture diagram of the system of the present invention; Figure 2 It is a flow chart of multi-modal monitoring data collection and processing inside the dam; Figure 3 It is a flow chart of external resource data collection and processing based on network search; Figure 4 This is the flow chart of dam historical data retrieval based on RAG; Figure 5 It is a dam hazard assessment flow chart based on BIM model, mathematical and physical model and their hybrid model; Figure 6 It is a flow chart of dam hidden danger propagation path analysis based on knowledge graph and graph database; Figure 7 is a model fine-tuning flow chart of the present invention; Figure 8 It is a flow chart of the comprehensive early warning and response mechanism for dam safety of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments to facilitate a clear understanding of the present invention, but they do not constitute a limitation on the present invention.

[0031] like Figure 1 As shown, the present invention provides a method for constructing and operating a multi-modal large model for reservoir dam safety, comprising the following steps: Constructing a training set: A single sample of the training set includes: multimodal data as model input: sensor data deployed on the dam, historical hidden danger records, engineering and technical documents formed at the planning, design, construction and operation stages of the dam, information resources related to dam safety, geometry and material information of the BIM model, timestamp and spatial location information; and whether a hidden danger event occurs and the probability value of the hidden danger event occurring as training labels; Use the training set to train the multimodal pre-trained model based on the Transformer architecture; Use real-time collected data to conduct real-time prediction and early warning of dam safety hazards based on trained models; Based on the probability value of whether the hidden danger event occurs output by the model, the actual impact degree of the hidden danger event, the comprehensive risk score, the warning threshold of the hidden danger event-related sensors and the emergency measures recommended are calculated.

[0032] The principle of the present invention is further explained below with reference to specific embodiments.

[0033] Example 1: A multi-modal large model construction and operation system for reservoir dam safety like Figure 1 As shown in the figure, a multi-modal large model construction and operation system for reservoir dam safety includes the following modules: Module 1: Multimodal dam internal monitoring data real-time perception module, which is used for collecting and processing multimodal monitoring data inside the dam, obtaining real-time and historical operating status data, and generating a comprehensive real-time monitoring view of the dam health status.

[0034] Monitoring data include physical quantities (deformation, seepage, stress, strain and temperature, etc.), images (photos, videos and point clouds, etc.), vibrations (strong earthquake monitoring, etc.), audio frequencies (metal structure monitoring), environment (meteorology, earthquake, hydrology, etc.), etc.

[0035] The real-time monitoring view of dam health status is a data visualization interface that comprehensively displays the current safety status and structural health of the dam. The real-time monitoring view of dam health status displays the following data: Real-time data overview: displays various real-time monitoring data of the dam, including stress, displacement, temperature, seepage and other physical quantities.

[0036] Status indicators: Health indicators calculated from various monitoring data, such as safety factor, risk level, etc., help quickly determine the overall health of the dam.

[0037] Anomaly detection: Real-time display of abnormal data or warning signals detected by sensors, helping managers to identify potential safety hazards in a timely manner.

[0038] Visualization charts: Use graphs and charts to intuitively display data change trends, such as stress change curves, displacement distribution diagrams, etc., to facilitate the identification of abnormal patterns. Regional heat maps: Use heat maps to show the health status of different parts of the dam, with color depth representing different risk levels or abnormalities in monitoring data.

[0039] Historical comparison: Provides comparison of historical monitoring data to help analyze changes between current and past states and identify long-term trends and potential problems.

[0040] Combining this information, the real-time monitoring view of the dam health status provides managers with a clear and intuitive tool to help them make quick and scientific decisions to ensure the safe operation of the dam.

[0041] Module 2: Real-time perception module of external resource data based on network search, which is used to obtain information resources related to dam safety around the world through automated web crawler technology.

[0042] Specifically, a large amount of external resource data is obtained through network search and data crawling; an inverted index and B+ tree index are established for the external resource data; index and parallel query technology are used to retrieve external resource data to find the target data; frequently accessed data sources and query results are cached; the target data and dam operation status data are weightedly fused and the weights are dynamically adjusted according to the data characteristics to form a comprehensive information resource data set.

[0043] Information resource data related to dam safety includes news reports on reservoir dam monitoring, safety assessment, monitoring and early warning, and maintenance and reinforcement around the world, as well as the latest academic papers, patents, laws, regulations, standards and other scientific and technological resources. These data are combined with multimodal sensor data to enhance the ability to assess the overall safety status of dams. The introduction of external resource data improves the comprehensiveness of decision-making and provides rich background information and reference for subsequent analysis modules.

[0044] Module three, the RAG-based historical data retrieval module, builds a knowledge document library based on the engineering and technical documents generated during the planning, design, construction and operation stages of the dam; divides the long documents in the knowledge document library into text blocks; uses a text embedding model to convert text blocks into digital vectors, captures the semantic information of the text, and forms a vector database; the vector database is used to retrieve text blocks related to the query and provide the most relevant information.

[0045] The dam knowledge document library covers a variety of important data types, including survey and design, construction records, daily dam operation data, hidden danger records, log records, hydraulic gate and turbine unit operation and maintenance records, and dam repair and reinforcement records, covering detailed information on all aspects of dam construction, operation, maintenance and repair.

[0046] To ensure that this information can be retrieved and used efficiently and accurately, the third step of the module first uses embedding technology to vectorize the historical data and convert the text information into digital vectors to facilitate subsequent calculations and comparisons. This vectorization not only retains the semantic information of the data, but also improves the efficiency of data processing.

[0047] During the retrieval process, after the user enters the query keywords, the system first vectorizes these keywords to generate a query vector. Then, the system uses the stored vectors in the vector database to find the historical data vector closest to the query vector through similarity calculation (such as cosine similarity). This process can quickly identify documents or records related to user needs.

[0048] To further improve the relevance of the search results, the system then uses reranking technology. Based on the initial search, this technology re-ranks the results and optimizes the sorting order of the search results by introducing more contextual information and features (such as the importance of the document, historical search frequency, etc.).

[0049] Specifically, reranking may use machine learning models to score preliminary results, taking into account the relevance of the document to the query, the quality of the document, and the user's historical preferences, to ensure that users can quickly find the most appropriate information when querying, thereby providing strong support for subsequent risk assessment and decision-making.

[0050] The knowledge document library supports subsequent data analysis and provides historical context for the system, helping to more accurately analyze current monitoring data. The historical data of the dam not only provides key references for the analysis of real-time monitoring data, but also adds necessary historical background to the assessment process, making the dam safety assessment more comprehensive and accurate.

[0051] Module 4: Construct a hidden danger assessment module based on BIM model, mathematical model, physical model and mathematical-physical hybrid model to evaluate the structural strength and stability of the dam based on real-time monitoring data and historical data, and obtain the location and type of hidden dangers.

[0052] Mathematical models, physical models and mathematical-physical hybrid models include finite element analysis models, fluid mechanics models, heat conduction models and soil mechanics models.

[0053] The finite element analysis model is used to identify stress concentration areas through finite element analysis; the fluid mechanics model is used to identify areas that may suffer structural damage due to excessive water pressure or water scouring through fluid mechanics analysis; the heat conduction model is used to identify areas where temperature changes affect structural safety through thermal stress distribution analysis; the soil mechanics model is used to identify areas where settlement or landslides may occur through soil mechanics analysis.

[0054] Specifically, Module 4 builds the BIM model of the dam based on engineering technical documents, and extracts attribute data from it to assign to each structural unit; associates historical operating status data, maintenance records, stress history and other information to the corresponding parts of the BIM model; based on externally acquired information resources, incorporates the latest technical standards, specifications, laws and regulations, accident cases and academic research into the BIM model, and updates risk assessment parameters and hidden danger identification models; Use the geometry and material information of the BIM model to conduct finite element analysis, fluid mechanics analysis, heat conduction analysis, and soil mechanics analysis to simulate the response of the dam under various working conditions and identify potential risk areas and features, including stress concentration areas, areas that may be subject to excessive water pressure or scour, areas that may crack due to excessive temperature stress, and foundation areas that may experience settlement or landslides; Compare the identified potential danger areas with the historical operating status data and historical potential danger records in engineering and technical documents to verify the accuracy of the model analysis results; and classify the risks of the identified historical potential dangers according to their characteristics; The identified potential danger areas will be highlighted in the BIM model and the potential danger characteristics will be marked; as new operating data and information resources are acquired, the BIM model and various analysis models will be continuously updated.

[0055] Module 4 integrates BIM (Building Information Modeling) models, mathematical models, physical models, and hybrid models of mathematical and physical models, combined with internal monitoring data, external resource data, and historical data to achieve dynamic assessment of dam safety hazards. The core of this module is to integrate multiple models to provide comprehensive and scientific analysis results to support dam safety management and decision-making. By integrating these models, the system can obtain the hidden dangers of the dam and the location of the hidden dangers.

[0056] Module 5: Hidden danger propagation path analysis module based on knowledge graph and graph database, which is used to construct knowledge graph and graph database, analyze the correlation between dam structure nodes according to real-time monitoring data and historical data to identify related hidden dangers, and realize the analysis and early warning of dam hidden danger propagation path, and obtain related hidden danger information and hidden danger propagation path.

[0057] Specifically, module five defines each structural unit of the dam as a structural unit node and assigns attributes; defines the physical connection and mechanical relationship between structural units as a structural connection edge and assigns attributes; Establish the relationship between nodes and edges: Establish the edges between the nodes of the structural units based on the historical operation status data, engineering technical documents and geometric information in the BIM model; Store the node and edge information in the graph database; calculate the nodes and edges in the graph database to obtain the node connectivity and importance score; Based on the real-time collected dam operation status data and new external information resources, the health status and risk level of the nodes in the knowledge graph, as well as the weight and impact probability of the edges, are updated; Using the updated node and edge attributes and the results of connectivity calculation, the path and probability of hidden dangers propagating from one structural unit to another are analyzed; Based on the probability of hidden danger propagation, a matrix representing the hidden danger correlation between nodes is constructed; the hidden danger impact matrix and graph database are combined to calculate the shortest path from the source node to other nodes of the hidden danger; according to the importance score of the node and its role in the hidden danger propagation process, the nodes with important influence in the hidden danger propagation process are identified.

[0058] Since the structure of the dam is composed of multiple interconnected components, an abnormality in any node may trigger a chain reaction, which in turn affects the safety of the entire structure. The knowledge graph represents the various structural units of the dam and their relationships in a graphical way, allowing the system to clearly display the connections and influences between nodes. As the underlying support of the knowledge graph, the graph database can efficiently store and query complex structural information. By abstracting the structural units of the dam into nodes in the graph, the attributes of each node not only contain basic physical information, but also integrate multi-dimensional data such as historical hidden danger records, detection data, and maintenance logs. This integration of information enables the system to quickly retrieve relevant nodes and analyze possible associated hidden dangers when hidden dangers occur.

[0059] In addition, by using the powerful path calculation capabilities of the graph database, the system can accurately analyze the path of hidden danger propagation. When an abnormality occurs at a certain node, the system can quickly calculate other nodes that the hidden danger may affect and identify the shortest path for the hidden danger to propagate.

[0060] By combining knowledge graphs with graph databases, we can not only systematically analyze the structural relevance of the dam, but also comprehensively identify potential associated hazards and their locations, providing more reliable data support and decision-making basis for ensuring dam safety. Through this multi-level analysis method, the dam monitoring system can achieve efficient linkage from early warning to decision-making, ensuring the overall health and safety of the dam.

[0061] Module six, pre-training and fine-tuning module of multimodal large model, pre-trains the multimodal large model by monitoring historical data, external data, historical information, hidden danger location and type, hidden danger propagation path, so that it can learn to obtain risk assessment results, early warning thresholds and early warning corresponding mechanisms based on the real-time monitoring data, external resource data, dam historical information, hidden danger location and type, and dam hidden danger propagation path, and fine-tune the multimodal large model in combination with the real-time monitoring data of the dam to realize intelligent analysis and adaptive optimization of multimodal data.

[0062] By pre-training and fine-tuning the multimodal large language model, a multimodal large language model for dam safety monitoring and assessment is obtained.

[0063] Specifically, during the model training process, the binary cross entropy loss function, gradient descent algorithm and L2 regularization are used to optimize the model parameters.

[0064] Before training and using the Transformer-based multimodal pre-training model, the input data of each modality is preprocessed through an independent encoder, and the features of each modality are fused through the Transformer's multi-head self-attention mechanism and used as model input.

[0065] In the training process of multimodal large language models, the construction of data sets is the key to the success of the system. In the data preprocessing stage, data cleaning, denoising and labeling are performed to ensure the accuracy and consistency of input data. Through high-quality data preprocessing, the training efficiency of the model is improved, and a solid foundation is laid for subsequent analysis and decision support.

[0066] In order to improve the model performance, this module uses fine-tuning technology to fine-tune the trained multimodal large language model so that the model can better adapt to the needs of dam monitoring.

[0067] Module seven: Comprehensive decision-making and early warning module based on multimodal large models.

[0068] This module is used for dam safety monitoring and assessment. It adopts a pre-trained multimodal large language model to monitor and assess dam safety. By acquiring real-time monitoring data, it obtains the probability of hidden dangers, the degree of impact of hidden dangers, and the warning threshold, thereby outputting risk assessment results and corresponding warning measures.

[0069] The actual impact of hidden danger events is calculated using the following formula: impact : ; Among them, I i represents the impact of the i-th hidden danger event on the dam, is the probability of occurrence of this type of hidden danger event, and n represents the number of hidden danger event types.

[0070] In the above technical solution, the comprehensive risk score R is calculated using the following formula:

[0071] Among them, α is the probability of hidden danger occurrence P failure The weight of (which can be calculated by weighted summation of the probability of occurrence of various hidden dangers) indicates its relative importance in risk assessment, and β is the weight of the impact of the hidden danger, indicating its contribution to the comprehensive score.

[0072] Specifically, the warning threshold of the hidden danger event associated sensor is comprehensively adjusted according to the following principles and their corresponding weights. The weight setting can be set according to human experience, and the final adjustment value is calculated according to the corresponding adjustment range and corresponding weight of each principle: When the probability of occurrence of any type of hidden danger is greater than its set upper threshold, the warning threshold of the sensor associated with the hidden danger is adjusted accordingly according to the extent by which the probability is greater than its set upper threshold, so as to improve the sensitivity of the sensor; when the probability of occurrence of any type of hidden danger is less than its set lower threshold, the warning threshold of the sensor associated with the hidden danger is adjusted accordingly according to the extent by which the probability is less than its set lower threshold, so as to reduce the sensitivity of the sensor; When the input dam safety-related information resource includes a weather forecast of a set type and degree, adjusting the warning threshold of a sensor associated with the weather forecast according to the weather forecast to improve the sensitivity of the sensor; Adjust the current warning threshold of the sensor according to the deviation between the historical warning threshold of the sensor and the historical hidden danger record; Adjust the current warning threshold of the sensor according to the abnormal fluctuation of the sensor.

[0073] Specifically, the process of determining emergency measures recommendations includes: when the probability of occurrence of any type of hidden danger is greater than its set upper limit threshold, the corresponding emergency measures recommendations for the hidden danger are obtained based on the technical standards and specifications contained in the input dam safety-related information resources and the historical maintenance and reinforcement records of the corresponding hidden dangers in the engineering technical documents.

[0074] Embodiment 2: A method for constructing and operating a multimodal large model for reservoir dam safety implemented by the multimodal large model construction and operation system for reservoir dam safety comprises the following steps: 1) The multi-modal sensor monitoring data collection and processing inside the dam is realized by the multi-modal dam internal monitoring data real-time perception block (module 1). Figure 2 As shown, the specific technical details and implementation methods are as follows: 1.1) Deployment and data collection of multimodal sensors The sensors include physical quantity sensors, vibration sensors, visual sensors, audio sensors, and environmental sensors to obtain multimodal data. Physical quantity sensors include deformation sensors, seepage sensors, stress strain sensors, temperature sensors, etc. Visual sensors include cameras, drones, etc. Environmental sensors include meteorological sensors, seismic sensors, hydrological sensors, etc.

[0075] Real-time data acquired by multimodal sensors are transmitted to the central processing system via wireless networks and stored in a unified database for data processing, data fusion and analysis of multimodal large models.

[0076] 1.2) Standardization and synchronization of multi-dimensional heterogeneous data (1) Standardization can eliminate the dimensional differences between data and ensure that data from different modalities can be compared and integrated in the same dimension. The formula for data standardization is: ; in, D i For i The raw data of each sensor, μ For the i The average value of sensor data, For the i The standard deviation of the sensor data, D norm For i The data of each sensor is normalized. After normalization, all data are converted into standardized data with a mean of zero and a standard deviation of one, thus ensuring the relative consistency between data from different sensors.

[0077] (2) Timestamp alignment: Use the spatiotemporal synchronization algorithm to align the timestamps of data from different sensors to ensure that all modal data are compared and analyzed in the same time dimension. The formula for time synchronization is: ; in, T sync is the timestamp after synchronization, T i is the timestamp of each sensor, nis the number of sensors. The algorithm ensures the consistency of the data in the time dimension, especially in the time synchronization between vibration, audio and visual data.

[0078] 1.3) Data fusion and error processing The data fusion of various sensors adopts the multi-modal weighted fusion method. The formula for data fusion is: ; in, D i From i The data of the sensors, w i For the i The system adjusts the weights of the sensor data. w i To optimize the fused data so that it can more accurately reflect the actual status of the dam.

[0079] Taking into account the measurement accuracy of different sensors and the errors that may be introduced by transmission delays. Therefore, the error calculation formula is used to evaluate the accuracy of data fusion E f : ; This formula is used to calculate the error value generated during the fusion process of different modal data. If the data of a certain sensor produces a large error after being fused with the data of other sensors, this module will dynamically adjust the data weight of the sensor. w i , to reduce errors and improve the accuracy of overall monitoring.

[0080] 2) Obtaining external resource data related to dam safety monitoring and assessment based on network search, which is realized by the real-time perception module of external resource data based on network search (module 2), including the collection and processing of external resource data based on network search, which is realized through automated network crawler technology. Figure 3 As shown in the figure, the external resource data collection and processing process based on network search, the specific technical details and implementation methods are as follows: 2.1) Network data acquisition, including the following main steps: (1) Index establishment and retrieval: Establish an inverted index structure and use a B+ tree index structure. The retrieved data is sorted based on factors such as relevance and timeliness to ensure that users get the most useful information. The ranking score formula is:

[0081] in, CTR is the click rate, Relevance For information P With queryQ The relevance of Freshness is the timestamp of information release, α, β, and γ are CTR, Relevance, Freshness Corresponding weights. Click-through rate (CTR) is obtained by recording the number of impressions of each search result and the number of clicks by users. When a user performs a search, the system automatically counts these data to calculate the CTR value. Relevance can be determined by manual annotation, user feedback, and machine learning models. Experts evaluate the relevance of each document and adjust it based on the user's click behavior and satisfaction feedback. In addition, machine learning models can automatically score by analyzing features. The acquisition of freshness depends on the timestamp of the record. The system evaluates the freshness of the information based on the difference between the current time and the time when the record was created. Through these methods, the system can comprehensively judge the quality and relevance of each search result. Through this ranking algorithm, the module can quickly obtain and utilize the latest information related to dam safety.

[0082] (2) Data caching and updating: Use a caching mechanism and regularly update the cached data. To improve retrieval efficiency, use a caching mechanism to cache frequently accessed data in local storage, such as specific standards and regulations, to reduce repeated queries; regularly update cached data to ensure the timeliness of the data.

[0083] (3) Parallel query and distributed computing: A distributed indexing solution is used to distribute external data across multiple servers or nodes for processing. When querying, tasks are assigned to multiple servers for parallel processing, thereby reducing the load pressure on a single server, reducing query response time, and further improving query efficiency.

[0084] 2.2) Network data fusion and application. By acquiring network data in real time, such as global patent information, international standards, laws and regulations, and the latest academic papers, the system can combine these external data with internal sensor data. First, external data provides the latest technical and normative references for dam safety assessment; second, sensor data provides real-time monitoring of dam health status. The system integrates these two types of data, analyzes potential risks and hidden dangers, and generates comprehensive assessment reports to support intelligent decision-making and ensure the safe operation and management of dams. For example: by acquiring global patent data in real time, the system can identify the latest monitoring and reinforcement technologies and help managers choose appropriate technical solutions for dam maintenance. By retrieving international standards and laws and regulations, the system can help dam operators adjust operating procedures, ensure compliance, and reduce potential risks caused by non-compliance. The latest academic papers and research results provide managers with rich background information to help conduct scientific risk assessment and decision-making.

[0085] 3) Obtaining historical data of dams, including building a knowledge base of basic dam data and performing efficient retrieval through vector database, which is realized by module 3. Figure 4 As shown, the specific technical details and implementation methods are as follows: 3.1) Build a knowledge base for basic dam information. The knowledge base for basic dam information includes dam survey, design and construction data, dam daily operation data, operation log records, fault log records, hidden danger records, turbine unit operation and maintenance records, hydraulic gate operation and maintenance data, dam repair and reinforcement records, etc. These documents cover detailed information on various aspects of dam construction, operation, maintenance and repair, and constitute an important basis for system analysis and decision support.

[0086] 3.2) Text segmentation. The documents in the dam basic information knowledge base are segmented into blocks to obtain text blocks. Text blocks represent different chapters, paragraphs or sentences of a document. The purpose of segmentation is to divide longer documents into smaller fragments for subsequent processing and calculation, and to improve processing speed and data operability while maintaining semantic integrity. Each text block contains meaningful semantic units, ensuring that the subsequent information extraction step can obtain high-quality and accurate content.

[0087] 3.3) Text embedding. The text block is processed by the text embedding model, and the text block is converted into a digital vector to obtain the text block vector. All the text block vectors constitute the vector database. The semantic information of the text is represented by the vector, and the vector database finally obtained is the knowledge base of the dam basic information.

[0088] Text embedding is achieved through machine learning algorithms. Text embedding models include Word2Vec, GloVe, and the BERT model based on deep learning. The model maps similar words or sentences into similar vector spaces by learning contextual relationships in a large-scale corpus. The closer two text blocks are semantically, the closer they are in the vector space. Through text embedding, the system can capture subtle differences and hidden semantic information in the text, providing a basis for information extraction and correlation analysis.

[0089] 3.4) Information extraction and retrieval based on vector database.

[0090] Information extraction and retrieval are achieved through the nearest neighbor search algorithm, which can quickly find the text block vector closest to the query vector and return the corresponding original text content. The nearest neighbor search algorithm includes the K nearest neighbor algorithm or the approximate nearest neighbor search algorithm. Law .

[0091] In practical applications, this function can quickly find records or knowledge related to the current problem from a large amount of documents. For example, if the system needs to check the maintenance records of a specific dam, it can extract the maintenance and repair data related to the dam from the dam basic information knowledge base for risk assessment or decision support.

[0092] 4) Based on the real-time monitoring data and historical dam data, dynamic assessment and analysis of dam safety hazards are carried out to determine the type and location of the hazards, which is achieved by module 4. Figure 5 As shown in the figure, the specific technical details and implementation methods of the dam safety hazard assessment process are as follows: 4.1) Construction of a refined BIM (Building Information Modeling) model of the dam. The BIM model uses three-dimensional visualization to represent the geometric information, material properties, construction history and maintenance records of the dam, providing accurate basic information for the analysis of mathematical models, physical models and mathematical-physical hybrid models.

[0093] For example, the system can use the BIM model to identify vulnerable parts of the dam or areas with a historical maintenance record. These areas are often the focus of assessment and may pose potential safety hazards.

[0094] 4.2) Analysis of dam safety hazards integrating mathematical models, physical models and mathematical-physical hybrid models: By combining the calculation results of BIM models and mathematical models, physical models and mathematical-physical hybrid models, a comprehensive safety assessment of the dam is conducted, and potential hazard areas, types and locations of hazards are accurately located.

[0095] Mathematical models, physical models and mathematical-physical hybrid models include finite element models, fluid mechanics models, heat conduction models and soil mechanics models.

[0096] The BIM model provides geometric information, material properties, historical records, and foundation data for the calculation of BIM models and mathematical models, physical models, and mathematical-physical hybrid models. The three-dimensional geometric structure of the BIM model is composed of multiple structural units. The geometric information (including volume, shape, and position) of each structural unit is discretized and transferred to the finite element model, fluid mechanics model, and heat conduction model for analysis to improve the accuracy of simulation calculations. Material properties include the strength, elastic modulus, and thermal conductivity of concrete, which are used for mechanical analysis of the dam. Historical records include the historical maintenance records and stress history information of the dam, which are used to identify areas where stress concentration has occurred or where repairs have been carried out, so as to better determine whether there are repeated hidden dangers in these areas.

[0097] Types of hidden dangers of dams include but are not limited to stress concentration, water scouring, cracks caused by thermal stress, and unstable foundation.

[0098] Stress concentration hazards: Finite element analysis models are used to simulate the stress and deformation distribution of the dam under different load conditions. The geometric information and material property data provided by the BIM model are discretized into multiple structural units, and the stress changes of these units under static loads (such as water pressure) and dynamic loads (such as earthquakes or floods) are calculated. Through stress tensor and deformation calculations, stress concentration areas are identified. These areas are often weak links in the structure and may cause cracks, deformations, or even failures. For example, when a part of the dam is subjected to excessive pressure or stress, the finite element analysis model can identify the location and, combined with the historical records of the BIM model, determine whether it is outside the safety threshold.

[0099] Water scour hazards: The fluid mechanics model is combined with the BIM model to analyze the dynamic impact of water flow on the dam. The fluid mechanics model simulates the interaction between water flow and the dam structure based on the dam body contour and boundary conditions provided in the BIM model, and predicts areas that may suffer structural damage due to excessive water pressure or water scour. The structural geometry information provided by the BIM model provides accurate boundary conditions for fluid mechanics analysis, allowing the fluid mechanics model to simulate real water flow behavior, thereby more accurately identifying water pressure sensitive areas.

[0100] Temperature change hazards: The heat conduction model simulates the thermal stress distribution of the dam under different temperature conditions, combined with the material properties and structural positions provided by the BIM model, to identify the expansion or contraction of materials due to temperature changes. Under extreme temperature conditions (such as extremely cold or hot climates), the dam material will produce thermal stress, resulting in cracks or deformation. By simulating and analyzing the stress distribution caused by temperature changes, the system can identify the affected parts and determine their stability.

[0101] Hidden dangers of unstable foundation: The soil mechanics model evaluates the stability of the dam foundation by analyzing the interaction between the dam and the foundation soil. The foundation data provided by the BIM model is combined with the soil mechanics model to evaluate the bearing capacity and sliding risk of the foundation. By analyzing the structural instability that may be caused by foundation settlement or sliding, potential hidden dangers of the dam foundation are identified, especially areas where settlement, landslides or collapses may occur, thereby ensuring the safety of the dam foundation. The common soil model is the Mohr-Coulomb model, which can evaluate the supporting effect of the soil on the dam and analyze the stability of the soil under extreme conditions.

[0102] 4.3) Hidden danger marking: mark the hidden dangers analyzed in the above models on the BIM model and determine the specific location of the hidden dangers in the dam structure.

[0103] The BIM model not only provides geometric and material information, but also helps to accurately locate the location of hidden dangers through its 3D visualization function.

[0104] Hidden danger marking: When mathematical, physical models and their hybrid models identify potential hidden dangers, these hidden dangers are marked on the corresponding structural elements of the BIM model. For example, when finite element analysis identifies a stress concentration area, the area is highlighted with a color on the BIM model to quickly identify the location of the hidden danger.

[0105] Hidden danger location: The specific location of hidden dangers in the dam structure is determined through the 3D BIM model. The 3D BIM model can view different areas of the dam from multiple perspectives and understand the spatial distribution of hidden dangers. Ensure the accurate location of hidden dangers to facilitate on-site maintenance and treatment.

[0106] 5) Based on the historical data and real-time monitoring data of the dam, the hidden dangers are identified and the propagation paths of the hidden dangers in the dam are traced, which is realized by module five. Figure 6 As shown in the figure, the dam hazard propagation path analysis process, the specific technical details and implementation methods are as follows: 5.1) Construct dam knowledge graph and graph database.

[0107] The structure of the dam is abstracted as a graph G = (V, E), where V It is a collection of nodes, representing the various structural units of the dam, including the dam body, spillway, foundation, etc. E It is a set of edges, which represents the connection relationship between structural units. Among them, the node contains the attribute information of the corresponding structure, including historical test data, material strength and stress distribution, etc. v The properties of are represented as a vector: A ( v ) = [historical test data, material strength, stress distribution]. Edges contain the physical connection of the structure and related attribute information. v With Node j Connected Edges e vj The properties are expressed as: W ( e vj )=[material strength, force transfer relationship]. The hidden danger propagation relationship is a dynamic attribute, which is closely related to the state of the node (such as the probability of hidden danger occurrence) and the propagation weight of the edge (such as the probability of hidden danger impact, the hidden danger propagation intensity). This type of information usually needs to be dynamically generated through real-time calculation or historical data analysis, rather than a fixed static attribute of the edge. Therefore, only static attributes (material strength and force transfer relationship) are listed in the initial definition. In this embodiment, the main purpose of constructing edge attributes is to describe the physical and mechanical relationships between structural units to support basic structural analysis (such as stress transfer). The hidden danger propagation relationship is more complex and may be divided into a separate calculation module instead of being the default static attribute of the edge.

[0108] The node set and edge set are stored in the graph database, which can efficiently manage the structural data and provide basic information for the subsequent hidden danger propagation analysis. This structured data storage method makes data query and analysis more efficient and can provide accurate information in a short time. The graph database contains node sets, edge sets, and node connectivity, which is used to indicate the degree of association between a structural unit and other structural units.

[0109] Node connectivity LJ ( v ) is used to measure the node v The relative importance of the node in the dam structure specifically reflects the degree of connection between the node and other structural units. v The connectivity of a node can be used to determine its position and role in the overall structure of the dam. If a node has a high connectivity, it means that it is connected to multiple other nodes, indicating that the node may have an important function or be subjected to greater stress in the structure. v Connectivity LJ ( v ) can be expressed as: ; in, A jv is an element in the adjacency matrix, representing a node j Is it related to the node v Connected, n is the total number of nodes in the knowledge graph.

[0110] The adjacency matrix is ​​a matrix used to represent the graph structure, which can clearly show the connection relationship between nodes. If the connectivity of a node is high, it means that the node is in a critical position in the dam structure and may have a greater impact on the overall safety of the dam.

[0111] The Pagerank algorithm is used to evaluate the relative importance of each node in the dam. The calculation formula is: ; in, PR ( v ) is a node v The relative importance score of B v For the node v The set of connected nodes, L ( u ) is a node u The number of outgoing edges, d is the damping factor (usually around 0.85), nis the total number of nodes. This score can help the system identify critical nodes in the dam structure, especially when the dam is under stress or hidden dangers, these critical nodes may be the parts that need the most attention.

[0112] 5.2) Identification of associated hazards and analysis of hazard propagation. Identify possible associated hazards near the hazard locations determined in step 4). The core of the knowledge graph is indeed composed of nodes and edges. Nodes represent entities (such as structural units of dams, hazards, etc.), while edges represent the relationships between nodes (such as connections, impacts, etc.). However, the richness and functionality of the knowledge graph are not only reflected in the connections between nodes and edges, but also include attribute information related to nodes and edges.

[0113] Node attributes: Each node can carry rich attribute information, such as historical test data, material strength, stress distribution, etc. These attributes provide important background for understanding the characteristics and status of the node. For example, the material strength and historical test data of a structural unit of a dam can help evaluate the safety of the unit.

[0114] Edge attributes: Edge attributes describe the relationship characteristics between nodes, such as force transmission relationship or connection strength. These attributes help analyze the mutual influence between different structural units, especially when hidden dangers occur, it is crucial to understand how forces are transmitted in the structure.

[0115] Information integration: Knowledge graphs can integrate information from different sources, such as historical hidden danger records, maintenance logs, and expert advice. This information can complement nodes and edges to help build a more comprehensive hidden danger network. For example, the location of a hidden danger may be related to a specific state of a node in the historical record, and this relationship can be described by an edge in the knowledge graph.

[0116] Hidden danger propagation analysis: By analyzing the characteristics of nodes and their attributes and edges, the knowledge graph can identify the associations between hidden dangers and trace the paths of hidden danger propagation within the dam. This propagation analysis not only relies on the connectivity of nodes and edges, but also comprehensively considers attribute information to more accurately assess the impact of hidden dangers on the entire structure.

[0117] Therefore, the value of the knowledge graph lies in its ability to provide a deep understanding and analysis of complex structures through rich attribute information of nodes and edges, thereby effectively supporting hidden danger identification and propagation analysis.

[0118] To analyze the spread of hidden dangers, the system constructs a hidden danger impact matrix. The hidden danger impact matrix is ​​a structured data table used to represent the hidden danger correlation between different structural units of the dam. In this matrix, rows and columns represent each node, and the value of each cell reflects the intensity or risk level of hidden danger propagation between two nodes. These values ​​can be calculated based on factors such as historical hidden danger records, material strength and stress distribution of nodes. By analyzing the hidden danger impact matrix, managers can identify high-risk nodes and formulate timely prevention and repair measures accordingly, thereby effectively reducing potential risks and ensuring the safe operation of the dam. By analyzing this matrix, managers can identify high-risk nodes and take timely measures.

[0119] The shortest path calculation based on the graph database is used to analyze the propagation path of hidden dangers in the dam structure. The calculation formula of the shortest path is: ; in, d ( u , v ) represents a node u and nodes v The shortest path distance between w It is on the path e i The weight of k is the number of edges on the path. In dam monitoring, the weight setting of edges is determined based on several key factors. First, physical distance is an important consideration. The shorter the distance between connected nodes, the lower the weight of the edge is usually set, because the potential danger is more likely to spread over a short distance. In addition, the strength of force transmission will also affect the weight. The higher the strength and stiffness of the connecting material, the greater the resistance to the spread of hidden dangers, and the corresponding weight can be lower. Historical hidden danger data is also important. If a path has often experienced hidden danger propagation in the past, the weight of the edge should be adjusted lower to reflect the higher risk. Environmental factors such as rainfall, temperature changes, and material properties also affect the weight setting. By comprehensively considering these factors, a reasonable weight can be set for each edge, making the shortest path calculation more accurate and ensuring the scientificity and effectiveness of the hidden danger propagation analysis. The possibility and speed of hidden danger propagation from one node to other nodes are calculated by the shortest path algorithm.

[0120] In addition, the graph database also supports complex path traversal queries, which can efficiently traverse the relevant paths in the dam structure according to the conditions set by the user, and provide detailed information on the path (such as the status of each node and the attributes of the edge). Path traversal query methods include using recursive traversal algorithms to query nodes v With Node u All paths between P ( u , v), and evaluates the importance of the path based on the edge weight w and node status on the path. In hidden danger propagation analysis, node status refers to the health or safety status of each node at a specific moment, reflecting its current safety and potential risks. Node status usually includes safety levels, such as "safe", "warning" or "dangerous", which is evaluated through real-time monitoring data such as stress, displacement and temperature. In addition, if a node has been identified as having hidden dangers, its status will include the type of hidden danger, severity and possible scope of impact. Historical monitoring records and maintenance logs will also affect the status of the node, especially nodes that have had hidden dangers. More attention is needed. Environmental factors, such as rainfall or earthquakes, can also affect the status of nodes, causing changes in their safety. By comprehensively evaluating the node status, the system can effectively predict the risk of hidden danger propagation and provide an important basis for management decisions. If a hidden danger occurs in a node, the system will calculate the impact of the hidden danger on its adjacent nodes and predict the possible scope of the hidden danger through the propagation model. The probability of hidden danger propagation can be expressed by the following formula: ; in, F ( u , v ) is a hidden danger slave node u Propagate to nodes v The probability of P ( u ) is a node u The probability of hidden dangers occurring, d ( u , v ) is a node u and nodes v The shortest path distance between d 0 is the influence radius set by the system. This formula shows that the possibility of hidden dangers spreading decreases as the distance between nodes increases. Based on the propagation prediction results, the system can trigger an early warning mechanism. Especially when key nodes are threatened, the system will automatically issue a warning to prompt dam managers to conduct further inspections or take preventive measures.

[0121] Get Node P ( u) The probability of hidden dangers can be achieved through a variety of methods. First, historical data analysis is an effective way to estimate the probability by counting the frequency of hidden dangers that occurred at the node in the past and the ratio of the total number of monitoring times. In addition, real-time monitoring data can also provide important information. The health status of the node can be evaluated by using data such as stress, displacement, and temperature monitored by sensors, combined with the set safety threshold, to determine the possibility of hidden dangers. Building a statistical model is also a common method to predict the probability of hidden dangers through a variety of influencing factors (such as material properties, environmental conditions, etc.). Finally, expert evaluation can provide a basis for probability and conduct in-depth analysis of node characteristics combined with professional knowledge. Through these methods, the system can dynamically update the probability of hidden dangers at the node and provide accurate support for the prediction of hidden danger propagation.

[0122] 5.3) The dam updates the knowledge graph and graph database. The data of nodes and edges in the knowledge graph and graph database are continuously updated incrementally as the monitoring data is updated, which is achieved by the incremental update algorithm.

[0123] The node status information is updated according to the real-time sensor data, including the structural health status, stress conditions, etc. At the same time, the attributes of the edges are also dynamically adjusted. For example, if the force transmission in certain areas is abnormal during structural detection, the weight of the corresponding edge is adjusted, and the shortest path and node importance are recalculated.

[0124] By introducing an incremental update algorithm, only the changed nodes and edges are updated, which greatly improves the system's response speed and operating efficiency.

[0125] 6) Pre-training and fine-tuning of the multimodal large model is implemented by module 6. The multimodal large model is pre-trained and fine-tuned to enable it to learn to obtain risk assessment results, warning thresholds and corresponding warning mechanisms based on the real-time monitoring data, external resource data, dam historical data, hidden danger locations and types, and dam hidden danger propagation paths. Figure 7 As shown in the figure, the technical details and implementation methods of pre-training and fine-tuning of the multimodal large model are as follows: 6.1) Dataset construction and preprocessing.

[0126] Dataset construction: collect historical multimodal monitoring data (from module 1), external resource data (from module 2), dam historical data (from module 3), hidden danger locations and types (from module 4), hidden danger propagation paths (from module 5), and form input samples based on the above data: sensor data deployed on the dam, historical hidden danger records, engineering and technical documents formed in the planning, design, construction and operation stages of the dam, information resources related to dam safety, geometry and material information of the BIM model, timestamp and spatial location information.

[0127] Data preprocessing: including data cleaning (removing invalid data, noise and outliers caused by sensor failure), format unification (such as equal-interval processing of time series data, vectorization of text data through word segmentation and keyword extraction), alignment and synchronization (time synchronization of multimodal data based on timestamps, and alignment of sensor data with BIM models through spatial coordinates) and normalization processing (Min-Max Scaling is used for continuous data, and one-hot encoding is performed on categorical data).

[0128] Data labeling: Create labels for each sample, including a binary label of whether a hidden danger event has occurred (0: not occurred, 1: occurred), the probability of hidden danger event occurrence (calculated by counting the ratio of the number of hidden danger occurrences under specific conditions to the total number of occurrences of the condition), and the type and impact of hidden danger events, as auxiliary labels for the event classification task. The calculation data of the data labels come from the historical data of the dam (from module 3), the location and type of hidden dangers (from module 4), and the propagation path of hidden dangers (from module 5).

[0129] Data augmentation: When data is insufficient or unbalanced, data augmentation techniques can be used to improve the generalization ability of the model, including cutting long time series into short time windows to extract more features, jointly sampling multimodal data to generate new samples, and using simulation methods to generate synthetic samples of sensor data or hidden danger propagation paths, thereby enriching the diversity and completeness of the data set.

[0130] Modal feature encoding: The Transformer model requires serialized vectorized feature input, so each modal data is feature encoded: sensor data uses sliding windows to extract time series features, visual data uses pre-trained CNN (such as ResNet or ViT) to extract image features, text data uses pre-trained language models (such as BERT) to generate embedded vectors, BIM model data extracts geometric and material properties and digitizes them, and spatiotemporal information encodes timestamps and spatial positions into numerical vectors. These features are processed separately by independent encoders to provide the model with a fusion basis for multi-modal features.

[0131] Data is organized as Transformer input format: The Transformer architecture requires that multimodal features be organized in sequence form. Each sample constitutes a sequence, in which each element corresponds to a feature vector of a modality (such as modality 1 features, modality 2 features, modality 3 features, ... modality 1 features, modality 2 features, modality 3 features, ... modality 1 features, modality 2 features, modality 3 features, ...). The fusion of modal features is achieved through the Transformer's multi-head self-attention mechanism, generating a joint representation with contextual relevance, which provides a basis for the comprehensive analysis of multimodal data.

[0132] Save training set: To facilitate efficient training and validation of the model, the training set needs to be saved in a format suitable for batch loading (such as .csv, .json or .tfrecord), and the data set needs to be divided according to certain rules. Usually, the data is divided into training set and validation set in chronological order or random proportion, for example, 80% of the data is used for training and 20% of the data is used for validation, so as to ensure the scientificity and accuracy of model training and evaluation.

[0133] 6.2) Model fine-tuning and pre-training Based on the above-established data set, the multimodal large model uses the Transformer's multi-head self-attention mechanism to fuse the features of each modality. These data from different sources are aggregated into the Transformer model for joint analysis, enabling the model to provide information support from multiple dimensions when capturing hidden dangers. Through the joint processing of multimodal data, the system can comprehensively assess the health of the dam based on abnormal audio signals combined with surface change information in images.

[0134] The loss function for pre-training the deep learning model based on the Transformer architecture is as follows: ; in, y i is the actual label, is the result predicted by the model, N is the total number of samples, and θ is the model parameter. This loss function is used to evaluate the difference between the model prediction results and the actual results.

[0135] In order to reduce the loss value of the model and optimize the parameters of the model, the gradient descent algorithm is introduced. The formula is as follows: ;

[0136] Among them, θ is the learning rate, is the gradient of the loss function to the parameter. By continuously updating the model parameters θ, the model is gradually optimized so that it can process and analyze the monitoring data of the dam more accurately.

[0137] In order to prevent the model from overfitting during training, L2 regularization technology is introduced to maintain the generalization ability of the model. Even when facing unseen data, the model can make accurate predictions. The formula is as follows: ;

[0138] Where λ is a regularization parameter, which is used to control the complexity of the model and prevent the model from being overly dependent on training data.

[0139] 6.3) Continuous learning and optimization of the model: Through online learning mode, the model is continuously updated with new monitoring data to maintain its sensitivity to environmental changes.

[0140] The key to online learning is the rapid response to new data and the dynamic adjustment of model weights. For example, when floods or extreme weather occur, the latest monitoring data is used to quickly update model parameters and improve the model's ability to respond to emergencies. At the same time, the system's closed-loop feedback mechanism allows each prediction result to be compared with the actual data, from which errors are obtained and the model is adjusted to maintain stable high performance during long-term monitoring.

[0141] In the process of continuous learning and optimization of the model, the error refers to the difference between the model prediction result and the actual observation value. This difference can be quantified in many ways. For example, the prediction error is the deviation between the model's output of the input data and the true label. The loss function is used to comprehensively evaluate this error, such as the commonly used loss functions such as mean square error and cross entropy. In online learning, the system compares each model prediction result with the latest monitoring data through a closed-loop feedback mechanism to calculate the feedback error. This process not only helps the model adjust weights and parameters, but also decomposes the error into deviation and variance to further optimize the model performance. Through continuous monitoring and analysis of errors, the system can improve its ability to respond to emergencies in real time and ensure scientific and effective decision-making support for dam safety management.

[0142] Through continuous optimization, the model weights and parameters are constantly adjusted according to the latest data to ensure that it can always provide scientific and effective decision-making support for dam safety management in various complex environments.

[0143] 7) Comprehensive dam safety early warning and response based on multimodal big model: Input the real-time monitoring data, the type and location of hidden dangers, the historical data of the dam related to the hidden dangers, the propagation path of the hidden dangers in the dam, and the external resource data related to the hidden dangers into the pre-trained multimodal big model, and analyze the dam historical data and real-time external resource data to obtain the probability of occurrence of hidden dangers in the dam, the degree of impact of hidden dangers, and the early warning threshold, so as to output the risk assessment results and the corresponding early warning mechanism, which is realized through module seven.

[0144] like Figure 8 As shown in the figure, the comprehensive early warning and response mechanism for dam safety based on the multi-modal large model, the specific technical details and implementation methods are as follows: 7.1) The multimodal large model predicts the probability of dam hidden dangers by integrating real-time monitoring data, external resource data, and historical data P failure and the possible impact of dam hazards C impact , thereby generating the final risk assessment results.

[0145] (1) Probability of hidden dangers occurring P failure : The multimodal large model uses real-time monitoring data from multimodal sensors, including water flow pressure, vibration, temperature, displacement and other information, to analyze the current structural status of the dam.

[0146] These data reflect the operation status of the dam under different environmental conditions and provide rich feature vectors for the multimodal big model. Through deep learning algorithms, the multimodal big model can identify potential hidden dangers and calculate their probability of occurrence. P failure In addition to real-time monitoring data from internal sensors, the multimodal big model also makes full use of external resource data and dam historical data to improve the accuracy of prediction. Through comprehensive analysis of these multi-dimensional data, the multimodal big model can more accurately determine the probability of hidden dangers caused by dam maintenance records and other factors, and dynamically adjust P failure The value of .

[0147] External resource data is obtained through online search, providing cutting-edge reference for risk prediction of multimodal big models. For example, accident case analysis around the world can help multimodal big models identify potential hazards similar to the current status of the dam, thereby adjusting the risk more accurately. P failure .

[0148] The historical data of the dam comes from the dam basic information knowledge base built by RAG (Retrieval Enhanced Generation). By analyzing the historical records, the multimodal big model can capture the long-term operation trend, maintenance history and hidden danger distribution of the dam, and combine these data to predict the probability of the current hidden danger. For example, if the historical records show that some areas have repeatedly had hidden dangers, the multimodal big model can increase the probability of hidden dangers in these areas. P failure value.

[0149] (2) Impact C impact :Multimodal large model evaluates the possible impact of dam hidden dangers C impact , assess the potential consequences of the risk. C impact It depends not only on the probability of occurrence of hidden dangers, but also on the degree of threat posed by the hidden dangers to the key parts of the dam. The multimodal large model calculates the possible damage caused by each type of hidden danger by analyzing historical hidden danger records and dam maintenance data. C impact The calculation formula is: ; Among them, Ii represents the impact of the i-th hidden danger event on the dam, is the probability of occurrence of this type of hidden danger event, and n represents the number of hidden danger event types. This formula integrates the risk levels of multiple hidden dangers and assigns different weights to each hidden danger. The model derives a comprehensive impact value based on factors such as structural damage, economic losses, and ecological impacts that these hidden dangers may bring. By quantifying the impact of various hidden dangers, managers can make preventive and response preparations in advance.

[0150] (3) Comprehensive risk score R： By calculating the probability of hidden dangers occurring P failure The degree of impact C impact Combined to derive a comprehensive risk score R :

[0151] in, α p The probability of hidden dangers occurring P failure The weight of β c For the degree of impact C impact Weight. α p and β c There are many ways to obtain the weights. First, the weight values ​​can be initially set with the help of expert experience and industry standards to ensure their rationality. Secondly, collect user feedback and the opinions of managers to understand the importance of the probability of hidden dangers and the degree of impact to decision-making in actual operations, so as to make corresponding adjustments. In addition, through A / B testing, different weight combinations can be used in different scenarios, their effects can be compared, and the best configuration can be found. Using machine learning models is also an effective method. By training the model, the weights are automatically optimized to make them more in line with actual needs. Finally, statistical analysis of historical data can help evaluate the changing trends of the probability of hidden dangers and the degree of impact, thereby providing data support for weight setting. Through these methods, the comprehensive risk score will be more accurate and can provide an effective decision-making basis for the safety management of the dam.

[0152] The output of the multimodal large model directly affects the comprehensive risk score R The final value of the dam can provide data support for the risk management of the dam. In this way, managers can have a more comprehensive understanding of the potential risks and formulate reasonable risk response strategies accordingly. R When the value is high, the system may recommend increasing the monitoring frequency, strengthening the maintenance of the dam, or preparing emergency measures.

[0153] The multimodal big model analyzes multimodal data to calculate the probability and impact of hidden dangers, providing accurate and intelligent support for dam safety risk assessment. The comprehensive risk score generated by the system not only helps identify potential hidden dangers, but also provides a scientific basis for dam managers to ensure timely and reasonable risk response strategies.

[0154] 7.2) Dam safety risk warning triggering. The multimodal large model dynamically adjusts the warning threshold by comprehensively analyzing internal sensor data, external resource data and historical data. The specific adjustment mechanism is as follows: (1) Threshold adjustment based on model output: When the multimodal large model predicts the probability of occurrence of a certain type of hidden danger P failure When the set threshold (for example, 70%) is exceeded, the system will actively lower the alarm threshold of the sensor related to the hidden danger to improve the sensitivity of monitoring. The degree of reduction is determined according to the degree to which the probability of the hidden danger exceeds the set threshold. For example, for every 5% increase in the probability of the hidden danger, the alarm threshold of the relevant sensor is reduced by 2% to 5%. Specifically, when the probability of hidden danger is high, the threshold is lowered; when the probability of hidden danger is low and the external environment is stable, the threshold can be moderately increased to reduce false alarms. For example, if the model predicts that the probability of structural stress hidden dangers in a certain area reaches 80%, which exceeds the set threshold by 10%, the system will reduce the alarm threshold of the stress sensor in the area by 5% to 10% to capture stress changes more sensitively.

[0155] (2) Threshold adjustment in response to external environmental data: When external environmental forecasts (such as heavy rain, earthquakes, etc.) show that the dam may have an impact, the system dynamically adjusts the alarm thresholds of relevant sensors according to the degree of environmental impact. When heavy rain is expected, the system lowers the alarm thresholds of seepage sensors and water level sensors. The reduction can be adjusted according to the rainfall level (moderate rain, heavy rain, heavy rain), for example, by 5% to 15%. In the case of earthquake warning, the system adjusts the alarm thresholds of ground stress and vibration sensors according to the magnitude of the earthquake and the distance between the epicenter and the dam. The greater the magnitude and the closer the distance, the greater the threshold reduction, which may be 10% to 20%. By lowering the threshold, the sensitivity of the sensor to anomalies caused by changes in the external environment is increased, ensuring that potential risks can be discovered in a timely manner.

[0156] (3) Threshold optimization based on historical data: The system regularly analyzes the deviation between sensor data and historical hidden danger records and dynamically adjusts the warning threshold. When the data of a certain sensor repeatedly triggers an alarm when it is 5% to 10% higher than the set threshold, the system determines that the current threshold may be too sensitive or insensitive. Depending on the degree of deviation, the threshold is appropriately raised or lowered, and the adjustment range is generally between 5% and 10%. For example, if historical data shows that the incidence of a certain type of hidden danger increases under specific environmental conditions (such as continuous high temperature), the system will lower the threshold of the relevant sensor during this period to improve monitoring sensitivity.

[0157] (4) Threshold anomaly detection and adaptive optimization: The system monitors changes in sensor data in real time and uses anomaly detection algorithms to dynamically adjust the warning threshold. When sensor data fluctuates abnormally (such as drastic changes in a short period of time) and does not conform to historical trends, the system identifies it as an anomaly. If the anomaly persists for more than a preset time (for example, 30 minutes), the system adjusts the threshold appropriately based on historical data and external information, with the adjustment range being 5% to 10%. When data anomalies are caused by equipment aging or external interference, the threshold is adjusted to avoid false positives or false negatives.

[0158] (5) Comprehensive assessment and threshold adjustment: The system comprehensively considers the probability of hidden dangers output by the model, changes in the external environment, and historical data to determine the direction and magnitude of threshold adjustment. In high-risk scenarios, when the probability of hidden dangers is high and the external environment is unfavorable, the threshold is lowered by up to 20%, and the monitoring frequency is increased. In low-risk scenarios, when the probability of hidden dangers is low and the environment is stable, the threshold can be moderately increased by 5% to 10% to reduce false alarms and improve system efficiency. The adjustment of the threshold is directly based on the probability of hidden dangers output by the model. The higher the probability, the more the threshold is lowered and the more sensitive the monitoring system is. This dynamic adjustment mechanism ensures the accuracy and flexibility of the early warning system.

[0159] 7.3) Dam safety risk early warning response mechanism. The dam safety risk early warning response mechanism is a key system to ensure that managers can quickly take effective countermeasures when facing potential risks. This mechanism relies on the output of a multimodal large model, combined with external resource data and dam historical data, to provide accurate and dynamic early warning response strategies for dam management. Through the network search function to obtain the latest global standards and specifications, academic papers and laws and regulations, as well as the dam basic information knowledge base built based on RAG technology, the system can establish a complete early warning response mechanism to ensure the scientificity and timeliness of decision-making.

[0160] The multimodal large model ensures that dam monitoring and emergency strategies meet international standards by analyzing the latest technical standards and specifications retrieved by module 2. For example, updates on emergency response to floods and earthquakes and installation requirements for monitoring equipment will be promptly incorporated into the early warning mechanism to help managers adjust emergency measures according to these standards to ensure their compliance and scientificity. Academic research provides the latest results for dam risk assessment, structural reinforcement, etc., supporting managers to formulate protection plans more scientifically and improve the effectiveness of emergency measures. In addition, the system can obtain the latest laws and regulations to ensure that dam managers comply with relevant laws in emergency responses such as disaster warning and evacuation to avoid legal risks.

[0161] The multimodal big model identifies high-risk areas based on historical maintenance and reinforcement records, and prioritizes monitoring these weak points in emergencies. Hidden danger logs and operation logs help the system analyze past hidden dangers and how they were handled, thereby providing response strategies for early warning responses. For example, if some areas have hidden dangers due to similar environmental conditions, the multimodal big model will propose corresponding emergency plans based on historical data, such as increasing monitoring frequency or implementing preventive reinforcement, to ensure that managers can respond quickly and reduce potential losses.

[0162] By combining external resource data obtained through online search functions with historical dam data provided by the RAG knowledge base, the multimodal large model ultimately forms a comprehensive risk warning response mechanism. The system can provide accurate early warning signals when the dam is facing risks, and combine standards and specifications, academic research, laws and regulations, and historical maintenance and reinforcement data to provide comprehensive emergency decision-making support for managers.

[0163] This mechanism ensures that managers can not only formulate emergency plans based on the latest standards, specifications, laws and regulations, but also identify high-risk areas and hidden danger transmission paths based on the historical operation data of the dam, and accurately formulate protection strategies. In an emergency, the system can dynamically adjust the response plan based on real-time analysis of multi-source data to ensure the scientificity, timeliness and compliance of emergency measures, and effectively ensure the safety and reliability of the dam.

[0164] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

Claims

1. A multi-modal large model construction and operation method for reservoir dam safety, characterized by: The following steps are involved: Constructing a training set: A single sample of the training set includes: multimodal data as model input: sensor data deployed on the dam, historical hidden danger records, engineering and technical documents formed at the planning, design, construction and operation stages of the dam, information resources related to dam safety, geometry and material information of the BIM model, timestamp and spatial location information; and whether a hidden danger event occurs and the probability value of the hidden danger event occurring as training labels; Use the training set to train the multimodal pre-trained model based on the Transformer architecture; Use real-time collected data to conduct real-time prediction and early warning of dam safety hazards based on trained models; Based on the probability value of whether the hidden danger event occurs output by the model, the actual impact degree of the hidden danger event, the comprehensive risk score, the warning threshold of the hidden danger event-related sensor and the emergency measure recommendations are calculated.

2. A method according to claim 1, characterized in that: After preprocessing, the data is used to construct the training set: the collected raw data of the training samples is cleaned to remove abnormal and invalid data; the data is annotated in a unified format and organized into structured data suitable for model training; The time alignment of image and vibration data is achieved through timestamps, and the spatial coordinates are uniformly processed.

3. A method according to claim 1, characterized in that: The sensors deployed on the dam include but are not limited to: physical quantity sensors, vibration sensors, visual sensor data, audio sensors and environmental sensors.

4. A method according to claim 1, characterized in that: During the model training process, the binary cross entropy loss function, gradient descent algorithm and L2 regularization are used to optimize the model parameters.

5. A method according to claim 1, characterized in that: Before training and using the Transformer-based multimodal pre-training model, the input data of each modality is preprocessed through an independent encoder, and the features of each modality are fused through the Transformer's multi-head self-attention mechanism and used as model input.

6. A method according to claim 1, characterized in that: The actual impact of hidden danger events is calculated using the following formula: impact : ； Among them, I i represents the impact of the i-th hidden danger event on the dam, is the probability of occurrence of this type of hidden danger event, and n represents the number of hidden danger event types.

7. A method according to claim 6, characterized in that: The comprehensive risk score R is calculated using the following formula: ； Among them, α is the probability of hidden danger occurrence P failure The weight of , indicating its relative importance in risk assessment, and β is the weight of the impact of hidden dangers, indicating its contribution to the comprehensive score.

8. A method according to claim 3, characterized in that: The warning thresholds of the sensors associated with potential hazards are adjusted comprehensively according to the following principles and their corresponding weights: When the probability of occurrence of any type of hidden danger is greater than its set upper threshold, the warning threshold of the sensor associated with the hidden danger is adjusted accordingly according to the extent by which the probability is greater than its set upper threshold, so as to improve the sensitivity of the sensor; when the probability of occurrence of any type of hidden danger is less than its set lower threshold, the warning threshold of the sensor associated with the hidden danger is adjusted accordingly according to the extent by which the probability is less than its set lower threshold, so as to reduce the sensitivity of the sensor; When the input dam safety-related information resource includes a weather forecast of a set type and degree, adjusting the warning threshold of a sensor associated with the weather forecast according to the weather forecast to improve the sensitivity of the sensor; Adjust the current warning threshold of the sensor according to the deviation between the historical warning threshold of the sensor and the historical hidden danger record; Adjust the current warning threshold of the sensor according to the abnormal fluctuation of the sensor.

9. A method according to claim 3, characterized in that: The process of determining emergency measures recommendations includes: when the probability of occurrence of any type of hidden danger is greater than its set upper limit threshold, the corresponding emergency measures recommendations for the hidden danger are obtained based on the technical standards and specifications contained in the input dam safety-related information resources and the historical maintenance and reinforcement records of the corresponding hidden dangers in the engineering technical documents.

10. A multi-modal large model construction and operation system for reservoir dam safety, characterized by: Including training set construction module, model training module, prediction module and early warning module: The training set construction module is used to construct a training set: a single sample of the training set includes: multimodal data as model input: sensor data deployed on the dam, historical hidden danger records, engineering and technical documents formed at the planning, design, construction and operation stages of the dam, information resources related to dam safety, geometry and material information of the BIM model, timestamp and spatial location information; And whether the hidden danger event as the training label occurs and the probability value of the hidden danger event; The model training module is used to train a multimodal pre-trained model based on the Transformer architecture using a training set; The prediction module is used to make real-time predictions and early warnings of dam safety hazards based on the trained model using real-time collected data; The early warning module is used to calculate the actual impact of the hidden danger event, the comprehensive risk score, the early warning threshold of the hidden danger event-related sensors, and emergency measures based on the probability value of whether the hidden danger event occurs output by the model.

Images