An image processing-based earth and rock dam hidden danger analysis system and cloud platform

Abstract

The application relates to the technical field of image processing, and particularly discloses a soil and rock dam hidden danger analysis system and cloud platform based on image processing, which comprises the following: a multi-modal data acquisition module and a data registration and fusion module are used to synchronously acquire visible light images, infrared thermal imaging data, laser radar point cloud data of a soil and rock dam surface, and view resistivity data and distributed optical fiber strain data reflecting the internal state of the soil and rock dam, and form a spatially aligned multi-source data field based on the acquired data; a cross-modal correlation analysis module is used to construct a final correlation feature map graph neural network model capable of outputting internal hidden danger probability; a physical constraint image recognition module is used to recognize and segment a two-dimensional hidden danger area segmentation mask on the surface of the soil and rock dam; and a hidden danger three-dimensional visualization and early warning module is used to trigger a hierarchical early warning signal based on a hidden danger evolution trend; the application solves the fundamental defects of a traditional soil and rock dam image analysis system, such as single data, missing correlation, mechanism disconnection and engineering disconnection of results.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing-based system and cloud platform for analyzing potential hazards in earth-rock dams. Background Technology

[0002] Currently, in the field of water conservancy engineering, earth-rock dams serve as crucial flood control and water storage facilities, and their safety and stability are directly related to the safety of people's lives and property and socio-economic development. An image processing-based earth-rock dam hazard analysis system and cloud platform, through in-depth analysis of relevant dam images and multi-source data, can accurately identify surface and potential internal hazards of the dam, providing a scientific basis for dam safety monitoring, hazard early warning, and maintenance decisions. With the advancement of water conservancy informatization and intelligentization, the application prospects of such systems and cloud platforms are broad, and they are expected to become a core technological support for ensuring the long-term safe operation of earth-rock dams, promoting the digital and intelligent transformation of dam safety management.

[0003] However, current technologies for analyzing potential hazards in earth-rock dams have many shortcomings. Traditional analysis systems often rely on single surface visible light images (such as drone aerial photography), which can only identify visible hazards exposed on the dam surface, resulting in blind spots and an inability to reach potential internal problems, leading to a superficial assessment of hazards. Even when some technologies employ multiple sensors, data silos often prevent precise physical correspondence between image processing results and internal state data (such as resistivity and strain data). Conclusions from different data sources are independent or even contradictory, making it difficult to form a unified hazard identification result. Furthermore, existing deep learning-based hazard identification models are mostly black-box models, highly dependent on training samples, lacking consideration of core physical mechanisms such as dam seepage and deformation. They have poor generalization ability for unfamiliar hazard types or complex conditions, easily leading to false alarms or missed alarms. In addition, traditional analysis outputs are mostly annotations on two-dimensional images, lacking true three-dimensional scale and spatial location information, making them difficult to use directly for engineering quantity assessment and precise maintenance decisions, resulting in a disconnect from actual engineering application needs.

[0004] Therefore, this invention proposes an image processing-based system and cloud platform for analyzing potential hazards in earth-rock dams. Summary of the Invention

[0005] This invention provides an image processing-based system and cloud platform for analyzing potential hazards in earth-rock embankments. By constructing a system architecture integrating multimodal data synchronous acquisition, unified spatial registration, cross-modal correlation analysis, physical mechanism constraint identification, and 3D visualization early warning, it systematically and hierarchically solves the fundamental defects of traditional earth-rock embankment image analysis systems, such as data homogeneity, missing correlations, disconnect from mechanisms, and results detached from engineering applications. The technical effect achieved is not merely an improvement in a single indicator, but rather a paradigm shift in this field from auxiliary surface observation to holistic paradigmatic support for mechanistic and chemical condition diagnosis and engineering decision support, constituting a complete, advanced technical solution with enormous application potential.

[0006] This invention provides an image processing-based system for analyzing potential hazards in earth-rock embankments, comprising: The multimodal data acquisition module is used to simultaneously acquire visible light images, infrared thermal imaging data, lidar point cloud data of the surface of the earth-rock embankment, as well as apparent resistivity data and distributed fiber strain data reflecting the internal state of the earth-rock embankment. The data registration and fusion module is used to establish a unified three-dimensional spatial coordinate system based on lidar point cloud data, and to register visible light images, infrared thermal imaging data, apparent resistivity data and distributed fiber strain data to the three-dimensional spatial coordinate system to form a spatially aligned multi-source data field. The cross-modal correlation analysis module is used to construct a graph neural network model. The graph neural network model takes a spatially aligned multi-source data field as input and learns the nonlinear correlation between surface texture features in visible light images, temperature anomaly features in infrared thermal imaging data, resistivity anomaly features in apparent resistivity data, and strain abrupt change features in distributed optical fiber strain data. It outputs the final correlation feature map that characterizes the probability of internal hidden dangers. The physical constraint image recognition module is used to encode the prior knowledge of seepage mechanics of earth-rock dams into constraint conditions and construct a deep semantic segmentation network that integrates the constraint conditions. The deep semantic segmentation network takes visible light images and associated feature maps as joint inputs, performs pixel-level classification, identifies and segments the cracks, seepage, and piping inlet hazard areas on the surface of earth-rock dams, and uses them as a two-dimensional hazard area segmentation mask. The 3D visualization and early warning module for hidden dangers is used to map the 2D hidden danger area segmentation mask back to the 3D spatial coordinate system, generate a 3D distribution model of hidden dangers in earth and rock embankments that integrates spatial location and monitoring time information, and trigger graded early warning signals based on the evolution trend of hidden dangers.

[0007] Preferably, the data registration and fusion module includes: The coordinate system submodule is used to establish a three-dimensional spatial coordinate system with real geographic coordinates or engineering coordinates by using the high-precision three-dimensional point cloud obtained by lidar scanning as a spatial reference. The multi-source registration and alignment submodule is used to match feature points in visible light images and infrared thermal imaging with lidar point clouds through feature extraction and matching algorithms, and to map the measurement point positions of apparent resistivity data and distributed fiber strain data to a three-dimensional spatial coordinate system through spatial interpolation algorithms to obtain a spatially aligned multi-source data field.

[0008] Preferably, the cross-modal correlation analysis module includes: The heterogeneous graph construction submodule is used to construct a heterogeneous graph data structure based on spatially aligned multi-source data fields. The nodes of the heterogeneous graph include image pixel nodes, resistivity measurement point nodes, and fiber strain measurement point nodes. The node feature vector contains the feature values ​​extracted from the corresponding data source. The edges of the heterogeneous graph are connected based on the similarity between the Euclidean distance of the nodes in the three-dimensional spatial coordinate system and the node feature vector, and are assigned initial weights. The associated feature learning submodule is used to input the constructed heterogeneous graph data structure into the graph neural network model. Through a multi-layer message passing mechanism, the node features are iteratively updated and aggregated, so that the final feature vector of each node is fused with the association information of cross-modal nodes in the corresponding neighborhood, and the final associated feature map representing the probability of internal hidden dangers is output.

[0009] Preferably, the associated feature learning submodule includes: The associated feature learning unit is used to input the constructed heterogeneous graph data structure into the graph neural network model. Through a multi-layer message passing mechanism, iterative updates and aggregation of node features are performed, so that the final feature vector of each node is fused with the association information of cross-modal nodes in the corresponding neighborhood, and the initial associated feature map representing the probability of internal hidden dangers is output. The confidence filtering unit is used to calculate the feature confidence value of each pixel position in the initial associated feature map and filter out isolated noise responses with confidence values ​​lower than a preset threshold to obtain the initial associated feature map after confidence filtering. The region fusion enhancement unit is used to fuse and smooth the discrete high-response regions based on spatial proximity and feature similarity in the initial association feature map after confidence filtering, generating a final association feature map with continuous spatial distribution and corresponding regional connectivity that matches the scale of typical physical anomaly regions inside the earth-rock dam.

[0010] Preferably, the physical constraint image recognition module includes: The knowledge encoding submodule is used to encode the prior knowledge of seepage mechanics of earth-rock dams into computable constraint rules, and generate physical constraint loss terms for training deep semantic segmentation networks based on the constraint rules. The prior knowledge of seepage mechanics of earth-rock dams includes constraints on the distribution range of the seepage line, constraints on the directionality of the seepage path, and constraints on the synergy between surface hazards and internal abnormal signals. The network training and application submodule is used to build and train a deep semantic segmentation network. The training loss function of the deep semantic segmentation network consists of pixel-level classification loss, region structure loss, and physical constraint loss. When the network is applied, the visible light image and the final associated feature map are concatenated as network input, and a pixel-level two-dimensional hazard region segmentation mask is output.

[0011] The preferred knowledge encoding submodule includes: The simulation model building unit is used to build a differentiable seepage simulation model, which is based on the parameterized dam geometry and the parameterized material permeability coefficient distribution field. The differentiable physics forward simulation unit is used to take the intermediate prediction results of the deep semantic segmentation network during the training process as internal defects or boundary condition parameters, input them into the differentiable seepage simulation model, perform a forward differentiable seepage field numerical simulation, and output the corresponding pore water pressure distribution cloud map and seepage velocity vector field. The physical field feature encoding unit is used to convert the pore water pressure distribution cloud map and seepage velocity vector field into physical field feature vectors through a differentiable encoding network; The Physical Consistency Loss Generation Unit is used to compare the physical field feature vector with the global feature vector extracted from the final association feature map corresponding to the same sample, which is output by the cross-modal association analysis module. The difference between the two in the feature space is calculated as a dynamic physical constraint loss term to optimize the deep semantic segmentation network.

[0012] Preferably, the differentiable seepage simulation model, the differentiable physical forward simulation unit, and the network training and application sub-module in the physical constraint image recognition module together constitute a physical information neural network training framework; The training process of the physical information neural network training framework executes the following steps sequentially within a single iteration cycle: The visible light image and the final associated feature map are input into a deep semantic segmentation network to perform image feature encoding and initial hazard area prediction. The initial hazard area prediction is mapped to the internal defect parameters of the differentiable seepage simulation model, which drives the differentiable seepage simulation model to perform seepage field simulation. The physical consistency of the physical quantity distribution of the seepage field output by the simulation is compared with the final associated feature map. The total training loss of the deep semantic segmentation network consists of pixel-level classification loss, physical constraint loss generated based on physical consistency comparison, and field reconstruction loss. The field reconstruction loss is used to measure the direct difference between the final associated feature map and the distribution of physical quantities of the seepage field output by the differentiable seepage simulation model. By using gradient backpropagation, the weight parameters of the deep semantic segmentation network and the material permeability coefficient parameters in the differentiable seepage simulation model are simultaneously optimized.

[0013] The preferred hazard 3D visualization and early warning module includes: The 3D modeling and mapping submodule is used to segment the 2D hidden danger area mask, and based on the spatial correspondence with the lidar point cloud, project it back into the 3D spatial coordinate system to construct a 3D solid model of the earth-rock dam hidden danger with geometric dimensions and spatial location, and bind it with temporal information. The dynamic risk assessment submodule is used to analyze the expansion speed and morphological change trend of hidden dangers based on the historical multi-period three-dimensional entity model sequence. Combined with real-time reservoir water level and rainfall operation parameters, the risk assessment model is used to calculate the risk level of the current hidden danger and the probability of its future evolution, and generate graded early warning information.

[0014] The preferred dynamic risk assessment submodule includes: The multi-scenario boundary initialization unit is used to generate a set of initial boundary conditions for multiple future scenarios, based on the three-dimensional distribution model of potential hazards in earth-rock dams identified at the current moment, combined with preset extreme working condition reservoir water level curves and rainfall process lines. The physical information cyclic deduction unit is used to input the corresponding set of initial boundary conditions into the physical information neural network training framework whose parameters have been optimized and converged for each deduction scenario. In the deduction mode, the weights of the fixed deep semantic segmentation network and the material parameters of the differentiable seepage simulation model are used as the input conditions for the next time step, along with the hazard segmentation results and seepage field state output by the framework of the previous time step. The coupling evolution process of the hazard in multiple consecutive time steps is simulated in an autoregressive cyclic iteration manner. The risk quantification unit is used to statistically analyze the simulation results of all cyclical scenarios, calculate the first occurrence time and cumulative probability of the hidden danger expanding to the critical geometric size or the local seepage gradient exceeding the soil's allowable seepage resistance strength, and generate graded early warning signals based on the corresponding probability timeline.

[0015] This invention provides a cloud platform for analyzing potential hazards in earth-rock dams based on image processing, comprising: Memory, used to store computer programs; A processor connected to memory is used to execute computer programs stored in memory to implement any of the above systems.

[0016] The beneficial effects of this invention compared to existing technologies are as follows: By fusing surface and internal data through a multimodal data acquisition module, it achieves comprehensive diagnosis from surface scanning to a holistic approach, not only identifying surface cracks but also revealing hidden hazards such as piping and seepage channels through correlation analysis, significantly expanding the types and stages of identifiable hazards. Using a unified three-dimensional spatial coordinate system for data registration, combined with graph neural network learning of cross-modal feature correlations, the accuracy of hazard location is improved to the real-world centimeter / decimeter level. Furthermore, cross-validation of multi-source data reduces false alarm rates and increases the detection rate of complex hazards, significantly enhancing identification reliability. The system incorporates prior knowledge of seepage mechanics in earth-rock dams as constraints, constructing a complete analysis chain, making the hazard identification results physically interpretable and enhancing engineers' confidence in the results. The generated three-dimensional hazard distribution model, integrating spatial location and temporal information, can be directly used to calculate hazard geometric parameters and assess their impact on dam stability, providing core data support for developing reinforcement plans and digital twin full lifecycle management, realizing the engineering practicality and value of the output results, and promoting a paradigm shift in this field from auxiliary observation to mechanistic diagnosis and decision support.

[0017] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained through the structures particularly pointed out in this application.

[0018] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a core technical architecture diagram of an image processing-based earth-rock dam hazard analysis system according to an embodiment of the present invention; Figure 2 This is a diagram illustrating the path of differentiable physics simulation technology in this embodiment of the invention. Detailed Implementation

[0020] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0021] like Figure 1 As shown, this invention provides an embodiment of an image processing-based system for analyzing potential hazards in earth-rock dams, comprising: The multimodal data acquisition module is used to simultaneously acquire visible light images, infrared thermal imaging data, lidar point cloud data of the surface of the earth-rock embankment, as well as apparent resistivity data and distributed fiber strain data reflecting the internal state of the earth-rock embankment. The data registration and fusion module is used to establish a unified three-dimensional spatial coordinate system based on lidar point cloud data, and to register visible light images, infrared thermal imaging data, apparent resistivity data and distributed fiber strain data to the three-dimensional spatial coordinate system to form a spatially aligned multi-source data field. The cross-modal correlation analysis module is used to construct a graph neural network model. The graph neural network model takes a spatially aligned multi-source data field as input and learns the nonlinear correlation between surface texture features in visible light images, temperature anomaly features in infrared thermal imaging data, resistivity anomaly features in apparent resistivity data, and strain abrupt change features in distributed optical fiber strain data. It outputs the final correlation feature map that characterizes the probability of internal hidden dangers. The physical constraint image recognition module is used to encode the prior knowledge of seepage mechanics of earth-rock dams into constraint conditions and construct a deep semantic segmentation network that integrates the constraint conditions. The deep semantic segmentation network takes visible light images and associated feature maps as joint inputs, performs pixel-level classification, identifies and segments the cracks, seepage, and piping inlet hazard areas on the surface of earth-rock dams, and uses them as a two-dimensional hazard area segmentation mask. The 3D visualization and early warning module for hidden dangers is used to map the 2D hidden danger area segmentation mask back to the 3D spatial coordinate system, generate a 3D distribution model of earth and rock dam hidden dangers that integrates spatial location and monitoring time information, and trigger graded early warning signals based on the evolution trend of hidden dangers.

[0022] In this embodiment, visible light images, infrared thermal imaging data, and lidar point cloud data of the earth-rock dam surface, as well as apparent resistivity data and distributed fiber optic strain data reflecting the internal state of the earth-rock dam, are acquired through a multimodal data acquisition module. Multiple data sources reflect the condition of the earth-rock dam from different perspectives: visible light images reveal surface texture, infrared thermal imaging data shows temperature anomalies, lidar point cloud data is used to construct spatial coordinates, and apparent resistivity and distributed fiber optic strain data reflect the internal state.

[0023] In this embodiment, the nonlinear correlation between surface texture features in visible light images, temperature anomaly features in infrared thermal imaging data, resistivity anomaly features in apparent resistivity data, and strain abrupt change features in distributed fiber optic strain data is as follows: In a scenario involving the monitoring of an earth-rock dam, when piping occurs within the dam, visible light images might reveal surface anomalies, such as subtle changes in texture on a previously smooth surface. This could be due to soil movement caused by the piping, resulting in surface texture features. In infrared thermal imaging, the temperature at the piping point will differ from the surrounding area due to heat exchange between the water flow and the surrounding soil, exhibiting temperature anomalies, possibly manifesting as localized temperature increases or decreases.

[0024] In apparent resistivity data, piping alters the pore structure and water content of the soil, causing changes in resistivity and exhibiting anomalies. For example, the resistivity near piping may significantly increase or decrease compared to normal areas. Distributed fiber optic strain data will show abrupt changes in strain sensed by the fiber within the piping's influence range. This is because soil deformation stretches or compresses the fiber, leading to sudden changes in strain data.

[0025] These features are not simply linearly related; they are intertwined and complexly interconnected. A change in one feature may trigger changes in other features, and the degree and manner of these changes are difficult to describe with a simple linear formula. This is the nonlinear relationship between them.

[0026] In this embodiment, the final correlation feature map representing the probability of internal hidden dangers is the output of the graph neural network model after learning the aforementioned nonlinear correlation relationship. It presents the probability distribution of hidden dangers existing inside the earth-rock dam in a visual form.

[0027] In this embodiment, constructing a graph neural network model is a crucial operation of the cross-modal correlation analysis module. A heterogeneous graph data structure is built through a heterogeneous graph construction submodule. Nodes include image pixels, resistivity measurement points, and fiber strain measurement points, etc. Edges are connected based on node spatial distance and feature vector similarity, and initial weights are assigned. The correlation feature learning submodule then inputs the heterogeneous graph into the graph neural network model. Node features are processed through a multi-layer message passing mechanism, outputting the final correlation feature map.

[0028] In this embodiment, prior knowledge of seepage mechanics of earth-rock dams is encoded into constraints. Seepage mechanics knowledge, such as the distribution range of the seepage line, the direction of the seepage path, and the synergy between surface hazards and internal abnormal signals, is transformed into constraint rules that can be used to train a deep semantic segmentation network, allowing the network to learn segmentation results that conform to physical laws.

[0029] In this embodiment, constructing a deep semantic segmentation network with integrated constraints means incorporating the aforementioned encoded constraints during the construction of the deep semantic segmentation network. This allows the network to combine knowledge of seepage mechanics in earth-rock dams during training and application, enabling it to more accurately identify potential hazard areas.

[0030] In this embodiment, the deep semantic segmentation network takes visible light images and associated feature maps as joint inputs, performs pixel-level classification, identifies and segments potential hazard areas such as cracks, seepage, and piping inlets on the surface of the earth-rock embankment, and uses these as a two-dimensional hazard area segmentation mask. After the network is trained, the visible light images and associated feature maps are used to perform pixel-level classification. Figure 1 Using the image as input, each pixel is classified to identify potential hazards such as cracks, leaks, and piping inlets, forming a two-dimensional hazard area segmentation mask that visually displays the location of the hazards.

[0031] In this embodiment, the two-dimensional hidden danger area segmentation mask is mapped back to the three-dimensional spatial coordinate system to generate a three-dimensional distribution model of earth and rock embankment hidden dangers that integrates spatial location and monitoring time information. Through the three-dimensional modeling and mapping sub-module, based on the spatial correspondence between the two-dimensional hidden danger area segmentation mask and the lidar point cloud, it is projected back to the three-dimensional spatial coordinate system, and monitoring time information is added to establish a model that can intuitively show the distribution of hidden dangers in three-dimensional space and their changes over time.

[0032] In this embodiment, triggering a graded early warning signal based on the evolution trend of the hidden danger means issuing graded early warning signals according to different probabilities and risk levels to remind relevant personnel to pay attention.

[0033] To establish a three-dimensional spatial coordinate system based on lidar point clouds, and to register and align multi-source data using feature extraction, matching, and spatial interpolation algorithms to form a spatially aligned multi-source data field, providing a unified spatial foundation for subsequent analysis, a data registration and fusion module is proposed, including: The coordinate system submodule is used to establish a three-dimensional spatial coordinate system with real geographic coordinates or engineering coordinates, using the high-precision three-dimensional point cloud obtained by lidar scanning as a spatial reference. The multi-source registration and alignment submodule is used to match feature points in visible light images and infrared thermal imaging with lidar point clouds through feature extraction and matching algorithms, and to map the measurement point positions of apparent resistivity data and distributed fiber strain data to a three-dimensional spatial coordinate system through spatial interpolation algorithms to obtain a spatially aligned multi-source data field.

[0034] In this embodiment, a high-precision 3D point cloud obtained by lidar scanning is used as a spatial reference to establish a 3D spatial coordinate system with real geographic coordinates or engineering coordinates. LiDAR scanning yields a large number of points representing the spatial location information of the earth-rock embankment surface, which together form a high-precision 3D point cloud. Using these point clouds as reference standards, and employing specific mathematical methods and techniques, a 3D spatial coordinate system with real geographic coordinates (such as latitude, longitude, and altitude, used for macroscopic positioning of the earth-rock embankment in geographic space) or engineering coordinates (a coordinate system set according to engineering design and construction requirements, used for precise positioning of various parts of the embankment within the engineering scope) can be constructed.

[0035] In this embodiment, feature extraction and matching algorithms (such as SIFT, VIMN, XoFTR, and FFM) are used to match feature points in visible light images and infrared thermal imaging with lidar point clouds. Spatial interpolation algorithms are then used to map the measurement point positions of apparent resistivity data and distributed fiber optic strain data to a three-dimensional spatial coordinate system, obtaining a spatially aligned multi-source data field. The feature extraction algorithm identifies representative and unique points, i.e., feature points, from the visible light images and infrared thermal imaging, such as corners on the surface of an earth-rock embankment or points with significant texture changes. Then, the matching algorithm finds corresponding points in the lidar point cloud, establishing a spatial connection between the visible light images, infrared thermal imaging, and lidar point clouds. For apparent resistivity data and distributed fiber optic strain data, each has its own measurement point positions. Using spatial interpolation algorithms, based on the established three-dimensional spatial coordinate system and the positional relationships of surrounding known points, the specific positions of these measurement points in the three-dimensional spatial coordinate system are calculated, thus integrating data from different sources into the same three-dimensional spatial coordinate system to form a spatially aligned multi-source data field.

[0036] In this embodiment, feature points in visible light images and infrared thermal imaging refer to points that can characterize specific locations or properties on the surface of an earth-rock dam. In visible light images, these feature points may be the endpoints of cracks on the surface of the earth-rock dam, points at the junctions of different materials, etc., reflecting the texture and structural information of the surface. In infrared thermal imaging, feature points may correspond to the boundary points of temperature anomaly areas, points at temperature abrupt changes, etc., and these points are related to the thermal characteristics of the earth-rock dam.

[0037] In this embodiment, the measurement point location for apparent resistivity data and distributed optical fiber strain data refers to the specific location where the measuring instruments are arranged when measuring the apparent resistivity and distributed optical fiber strain inside the earth-rock dam.

[0038] To construct heterogeneous graphs from spatially aligned multi-source data fields, a multi-layer message passing mechanism of a graph neural network is used to iteratively update the features of aggregated nodes, outputting a final association feature map representing the probability of internal hidden dangers. This enables effective association analysis of cross-modal data. A cross-modal association analysis module is proposed, including: The heterogeneous graph construction submodule is used to construct a heterogeneous graph data structure based on spatially aligned multi-source data fields. The nodes of the heterogeneous graph include image pixel nodes, resistivity measurement point nodes, and fiber strain measurement point nodes. The node feature vector contains the feature values ​​extracted from the corresponding data source. The edges of the heterogeneous graph are connected based on the similarity between the Euclidean distance of the nodes in the three-dimensional spatial coordinate system and the node feature vector, and are assigned initial weights. The associated feature learning submodule is used to input the constructed heterogeneous graph data structure into the graph neural network model. Through a multi-layer message passing mechanism, the node features are iteratively updated and aggregated, so that the final feature vector of each node is fused with the association information of cross-modal nodes in the corresponding neighborhood, and the final associated feature map representing the probability of internal hidden dangers is output.

[0039] In this embodiment, a heterogeneous graph data structure is constructed based on spatially aligned multi-source data fields. This means that the structure is built upon multi-source data such as visible light images, infrared thermal imaging data, apparent resistivity data, and distributed fiber optic strain data, which are already aligned in a three-dimensional spatial coordinate system. Different types of data are mapped to different types of nodes, thereby constructing a special data structure with diverse node types, which is called a heterogeneous graph data structure.

[0040] In this embodiment, image pixel nodes, resistivity measurement point nodes, and fiber optic strain measurement point nodes are different types of nodes in a heterogeneous graph data structure. Image pixel nodes correspond to pixels in a visible light image, with each node representing a small region in the image and reflecting the image characteristics of that region. Resistivity measurement point nodes correspond to the measurement point locations of apparent resistivity data, reflecting information related to the resistivity characteristics of the soil and rock at that location. Fiber optic strain measurement point nodes correspond to the measurement points of distributed fiber optic strain data, used to reflect the strain conditions inside the dam at that point.

[0041] In this embodiment, the node feature vector contains feature values ​​extracted from the corresponding data source. For each type of node, representative information is extracted from its corresponding data source to form a vector. For example, feature values ​​such as color and texture are extracted from the region corresponding to the visible light image pixel node to form the feature vector of the image pixel node; feature values ​​such as resistivity magnitude and change trend are extracted from the measurement data corresponding to the resistivity measurement point node to form the feature vector; and feature values ​​such as strain magnitude and change direction are extracted from the strain measurement data to form the feature vector of the fiber optic strain measurement point node.

[0042] In this embodiment, the edges of the heterogeneous graph are constructed based on the Euclidean distance between nodes in the three-dimensional coordinate system and the similarity of node feature vectors, and are assigned initial weights. In the three-dimensional coordinate system, the Euclidean distance between different nodes is calculated, and the similarity of node feature vectors is compared. If two nodes are spatially close and have high feature vector similarity, they are connected by an edge. These edges are assigned initial weights, the magnitude of which reflects the tightness of the connection between the two nodes; edges between nodes that are close and have high feature similarity have relatively larger weights.

[0043] In this embodiment, the constructed heterogeneous graph data structure is input into the graph neural network model. A multi-layer message passing mechanism iteratively updates and aggregates node features, enabling each node's final feature vector to incorporate the association information of cross-modal nodes within its neighborhood. This results in the output of a final association feature map representing the probability of internal hazards. The model employs a multi-layer message passing mechanism; in each layer, nodes transmit some of their own feature information to their neighboring nodes while also receiving information from neighboring nodes. Through multiple such information transmissions and fusions, nodes continuously update their own features. Aggregation refers to nodes integrating the received information from neighboring nodes into their own feature vectors. Ultimately, each node's feature vector not only contains features from its own data source but also incorporates the association information of different types (cross-modal) nodes within its neighborhood. In this way, the model outputs a final association feature map representing the probability of internal hazards within the earth-rock dam, aiding in the assessment of internal hazard conditions.

[0044] In this embodiment, the multi-layer message passing mechanism is a method used for information interaction between nodes in a graph neural network model. During each layer of message passing, a node sends messages (i.e., partial feature information of itself) to neighboring nodes based on its own characteristics and its connection relationships with neighboring nodes (edge ​​weights), while simultaneously receiving messages from neighboring nodes. Through multiple layers of such message passing, nodes continuously update their own features, making them richer and more comprehensive, reflecting their relationships with surrounding nodes and overall structural information.

[0045] In this embodiment, node features refer to the set of information describing the characteristics of each node, represented by a node feature vector composed of feature values ​​extracted from the corresponding data source. For example, image pixel node features may include color and texture features, while resistivity measurement point node features may include resistivity-related features, etc.

[0046] In this embodiment, the final feature vector of a node is obtained by continuously updating and aggregating information from neighboring nodes through a multi-layered message passing mechanism. It is a comprehensive feature vector that integrates cross-modal node association information within the neighborhood. This vector can more comprehensively reflect the characteristics of the node in the entire heterogeneous graph data structure and its relationship with other nodes.

[0047] In this embodiment, the association information of cross-modal nodes within a neighborhood refers to the information carried by nodes adjacent to a certain node (connected by edges) but belonging to different data source types (cross-modal), as well as their relationship information with that node. For example, the feature information of resistivity measurement points and fiber strain measurement points within the neighborhood of an image pixel node, and the degree of connection between these different types of nodes as reflected by the edge weights. This association information is fused into the final feature vector of the node through a multi-layer message passing mechanism.

[0048] To learn cross-modal association information through graph neural networks to generate an initial association feature map, noise is removed by confidence filtering, and discrete regions are enhanced by fusing spatial proximity and feature similarity to generate a final association feature map adapted to the scale of physical anomalies, thereby improving the quality of the association feature map, an association feature learning submodule is proposed, including: The associated feature learning unit is used to input the constructed heterogeneous graph data structure into the graph neural network model. Through a multi-layer message passing mechanism, iterative updates and aggregation of node features are performed, so that the final feature vector of each node is fused with the association information of cross-modal nodes in the corresponding neighborhood, and the initial associated feature map representing the probability of internal hidden dangers is output. The confidence filtering unit is used to calculate the feature confidence value of each pixel position in the initial associated feature map and filter out isolated noise responses with confidence values ​​lower than a preset threshold to obtain the initial associated feature map after confidence filtering. The region fusion enhancement unit is used to fuse and smooth the discrete high-response regions based on spatial proximity and feature similarity in the initial association feature map after confidence filtering, generating a final association feature map with continuous spatial distribution and corresponding regional connectivity that matches the scale of typical physical anomaly regions inside the earth-rock dam.

[0049] In this embodiment, the constructed heterogeneous graph data structure is input into a graph neural network model. A multi-layer message passing mechanism iteratively updates and aggregates node features, enabling the final feature vector of each node to fuse the association information of cross-modal nodes within its corresponding neighborhood. This outputs an initial association feature map representing the probability of internal hidden dangers. Essentially, the constructed heterogeneous graph data structure, containing different types of nodes (such as image pixel nodes, resistivity measurement point nodes, etc.) and their interconnections, is fed into the graph neural network model. The model utilizes a multi-layer message passing mechanism to repeatedly transmit and integrate information between nodes, ensuring that the final feature vector of each node includes relevant information from neighboring nodes of different types. Based on these fused node features, the model generates a graph that reflects the possibility of hidden dangers within the earth-rock dam; this is the initial association feature map.

[0050] In this embodiment, the pixel positions in the initial associated feature map refer to the location of each pixel, which, like a regular image, is composed of individual pixels. By analyzing the pixel positions, the locations of the earth-rock embankments represented by different regions in the image can be determined.

[0051] In this embodiment, calculating the feature confidence value for each pixel in the initial associated feature map is to measure the reliability of the information represented by each pixel in the initial associated feature map in characterizing the probability of internal hidden dangers. Through a specific algorithm, considering factors such as the features of surrounding nodes and the relationship with other pixels, each pixel is assigned a value; this value is the feature confidence value. The higher the value, the more reliable the internal hidden danger probability information reflected by that pixel. Assume the initial associated feature map is a simple two-dimensional matrix representing a certain area of ​​an earth-rock dam. A simplified algorithm is used to calculate the feature confidence value.

[0052] Taking a pixel P as an example, first consider the node features within its 3×3 neighborhood (assuming these node features are simple numerical values ​​extracted from corresponding data sources, such as grayscale values, temperature values, etc.). Calculate the average value A of all node features within this neighborhood, and the variance V of these feature values.

[0053] At the same time, the grayscale differences D1, D2, D3, and D4 between pixel P and its neighboring pixels (such as the four directly adjacent pixels in the top, bottom, left, and right directions) are analyzed.

[0054] Then, the feature confidence value C is calculated using the formula: C = α × A - β × V + γ × (D1 + D2 + D3 + D4) / 4. Here, α, β, and γ are weighting coefficients set according to the actual situation, such as α = 0.5, β = 0.3, and γ = 0.2.

[0055] For example, the calculated feature confidence value of pixel P is 0.7, which is relatively high, indicating that the internal hidden danger probability information it reflects is relatively reliable. This algorithm comprehensively considers the features of surrounding nodes and the relationship with neighboring pixels to assign a feature confidence value to each pixel.

[0056] In this embodiment, the preset threshold is a pre-defined standard value. This value is used to compare with the feature confidence value of each pixel location in the initial associated feature map, serving as the basis for determining whether the pixel information is reliable and whether it needs to be retained. The preset threshold can be set to 0.7.

[0057] In this embodiment, isolated noise responses with confidence levels below a preset threshold are filtered out. After calculating the feature confidence value for each pixel location, pixels with confidence levels below the preset threshold are identified as responses that may be generated by noise. Furthermore, if these pixels are isolated, meaning there are no surrounding pixels with similar confidence values, they are removed from the initial associated feature map.

[0058] In this embodiment, discrete high-response regions refer to those areas in the initial correlation feature map after confidence filtering that have high feature confidence values ​​but are spatially separated and discontinuous. These regions may indicate a high probability of hidden dangers within the earth-rock dam, but because they are discrete, further processing is required to more accurately reflect the situation of the hidden dangers.

[0059] In this embodiment, the initial association feature map after confidence filtering is fused and its boundaries smoothed based on spatial proximity and feature similarity. This generates a final association feature map with continuous spatial distribution and connectivity that matches the scale of typical physical anomaly areas within the earth-rock dam. Essentially, these discrete high-response regions are merged based on their spatial proximity and the similarity of the features they represent. Simultaneously, the boundaries of the merged regions are smoothed to avoid abrupt boundaries.

[0060] In this embodiment, the continuous spatial distribution and the connectivity of corresponding regions are adapted to the scale of typical physical anomaly areas inside the earth-rock dam. This means that the areas representing potential hazards in the final associated feature map are spatially continuous, without obvious breaks or gaps. Furthermore, the connection methods and sizes of these areas correspond to the connection methods and sizes of areas corresponding to actual physical anomalies (such as piping and cracks) that may occur inside the earth-rock dam. This allows for a more accurate assessment of the actual situation of potential hazards inside the dam from the map.

[0061] In this embodiment, the scale of typical physical anomaly areas inside an earth-rock dam refers to the size range of areas formed by common physical anomalies such as piping, cracks, and seepage within the earth-rock dam. For example, the length and width range of a typical crack, and the diameter range of the piping formation area.

[0062] In this embodiment, the spatial distribution is continuous, and the corresponding regional connectivity is also important. The continuous spatial distribution means that the areas potentially posing risks in the final associated feature map are spatially continuous and uninterrupted, without isolated small areas scattered in the distance. The corresponding regional connectivity emphasizes that these potentially posing risk areas are interconnected, and the connection method is similar to the connection method between physically abnormal areas inside an actual earth-rock dam, reflecting the inherent connection between risky areas.

[0063] To encode prior knowledge of seepage mechanics in earth-rock dams into constraint rules to generate physical constraint loss terms, a deep semantic segmentation network incorporating these loss terms is constructed and trained to achieve pixel-level identification and segmentation of potential hazard areas based on physical constraints. A physical constraint image recognition module is proposed, including: The knowledge encoding submodule is used to encode the prior knowledge of seepage mechanics of earth-rock dams into computable constraint rules, and generate physical constraint loss terms for training deep semantic segmentation networks based on the constraint rules. The prior knowledge of seepage mechanics of earth-rock dams includes constraints on the distribution range of the seepage line, constraints on the directionality of the seepage path, and constraints on the synergy between surface hazards and internal abnormal signals. The network training and application submodule is used to build and train a deep semantic segmentation network. The training loss function of the deep semantic segmentation network consists of pixel-level classification loss, region structure loss, and physical constraint loss. When the network is applied, the visible light image and the final associated feature map are concatenated as network input, and a pixel-level two-dimensional hazard region segmentation mask is output.

[0064] In this embodiment, encoding prior knowledge of seepage mechanics in earth-rock dams into computable constraint rules involves transforming existing professional knowledge in the field of seepage mechanics in earth-rock dams into rules that computers can understand and process through specific methods. For example, knowledge about how seepage flows within the dam and the appropriate range of the phreatic line is expressed using mathematical formulas and logical judgments, enabling computers to analyze and process relevant data based on these rules. For instance, if the knowledge of seepage mechanics in earth-rock dams states that the phreatic line should normally be within 10-20 meters of the dam crest, the specific method would be to encode this knowledge using logical judgments and mathematical formulas.

[0065] First, define several variables, such as using the seepage line position to represent the actual measured distance between the seepage line and the dam crest. Then, write a rule like this: if the "seepage line position" is less than 10 meters or greater than 20 meters, it is considered an anomaly. This is using logical judgment to express the knowledge of the seepage line range.

[0066] For example, seepage follows Darcy's law, which states that the seepage velocity equals the permeability coefficient multiplied by the hydraulic gradient. In a computer, the permeability coefficient and hydraulic gradient are treated as available numerical variables. By writing formulas to calculate the seepage velocity, such as: seepage velocity = permeability coefficient × hydraulic gradient, this is expressing the knowledge of seepage flow using mathematical formulas.

[0067] In this embodiment, a physical constraint loss term for training the deep semantic segmentation network is generated based on constraint rules. A calculation method is designed to generate this loss term based on the previously encoded computable constraint rules. This loss term is used in the training process of the deep semantic segmentation network; it measures the difference between the network's predictions and prior knowledge of seepage mechanics. When the network's predictions do not conform to this prior knowledge, the value of the loss term increases, prompting the network to adjust its parameters to make the predictions more consistent with actual seepage mechanics. Suppose that prior knowledge of seepage mechanics encodes a constraint rule: the seepage line should be within 10-20 meters of the dam crest. The deep semantic segmentation network predicts the seepage line location to be 25 meters from the dam crest.

[0068] The calculation method can be designed as follows: First, calculate the difference between the predicted value (25 meters) and the boundary value of the reasonable range (10-20 meters). Subtracting 20 from 25 gives 5. Take the absolute value of the difference, which is 5. This 5 is the value of the loss term. If the predicted value is within the range of 10-20 meters, the difference is 0, and the loss term is 0.

[0069] In this way, the more the predicted infiltration line position deviates from the reasonable range, the larger the loss term becomes. During network training, the parameters will be adjusted based on this larger loss term so that the next prediction is more in line with the prior knowledge of the infiltration line position.

[0070] In this embodiment, the constraints on the distribution range of the seepage line, the directionality of the seepage path, and the synergistic constraint between surface hazards and internal abnormal signals are specific constraint rules formed by encoding prior knowledge of seepage mechanics. The constraint on the distribution range of the seepage line specifies the range within which the seepage line should be located under normal conditions within the earth-rock dam. If the network predicts the seepage line location beyond this range, it does not conform to reality. The constraint on the directionality of the seepage path clarifies the direction that seepage should follow within the dam, such as flowing from high head to low head; the predicted seepage path direction must conform to this constraint. The synergistic constraint between surface hazards and internal abnormal signals requires that when a hazard such as a crack appears on the surface, the corresponding internal signals such as apparent resistivity and strain should also show corresponding anomalies; the two must correspond to each other.

[0071] In this embodiment, constructing and training a deep semantic segmentation network involves building a neural network with a specific structure. This network can perform semantic understanding on the input image data, segmenting different regions in the surface image of an earth-rock embankment according to their semantics (such as cracks, seepage areas, etc.). After constructing the network structure, a large amount of earth-rock embankment image data and corresponding labels (marking the actual potential hazard areas in the images) are used to train the network, adjusting the network parameters to enable it to accurately identify and segment potential hazard areas. The construction process includes: First, determine the network architecture, such as using a U-Net-like structure, which has an encoder and a decoder. The encoder is responsible for gradually reducing the image size and extracting features, for example, by using successive convolutional and pooling layers to shrink the image from large to small, while extracting key features. The decoder, on the other hand, uses upsampling and convolutional operations to gradually restore the segmentation result to the same size as the original image by using the features extracted by the encoder. In this process, different features are combined to identify different semantic regions. Skip connections can also be added in the middle, allowing shallow features from the encoder to be directly connected to the corresponding layers in the decoder, helping to recover details. This initially constructs a deep semantic segmentation network, ready for subsequent training.

[0072] In this embodiment, the training loss function of the deep semantic segmentation network is used to evaluate the difference between the network's predictions and the actual results during training. By continuously adjusting the network parameters, the value of this loss function is minimized, thereby making the network's predictions as close to the reality as possible.

[0073] In this embodiment, the training loss function of the deep semantic segmentation network consists of pixel-level classification loss, region structure loss, and physical constraint loss. Pixel-level classification loss measures the accuracy of the network's classification of each pixel, i.e., the accuracy in determining whether each pixel belongs to a potential hazard area. Region structure loss focuses on the degree to which the segmented potential hazard areas match the actual potential hazard areas in terms of shape, size, and spatial structure. The physical constraint loss term ensures that the network's prediction results conform to the aforementioned prior knowledge of seepage mechanics.

[0074] Suppose we have an image of an earth-rock dam and its corresponding annotations (the actual potential hazard areas).

[0075] Pixel-level classification loss: The probability that each pixel is a potential danger area predicted by the network is compared with whether the pixel is actually a potential danger area in the label. For example, the cross-entropy loss function can be used. The greater the difference between the predicted probability and the actual situation, the greater the loss. This is how the pixel-level classification loss is determined.

[0076] Regional structural loss: The differences between the predicted hazard area and the actual hazard area in terms of shape (e.g., aspect ratio), size (area), and spatial structure (e.g., adjacency relationship) are calculated. The greater the difference, the greater the loss, thus determining the regional structural loss.

[0077] Physical constraint loss term: Based on prior knowledge of seepage mechanics, such as the wetting line range and seepage direction, the degree of conformity between the network prediction results and these prior knowledge is judged. The greater the degree of non-conformity, the greater the loss, and thus the physical constraint loss term is determined.

[0078] In this embodiment, when the network is applied, the visible light image and the final associated feature map are concatenated in channels as network input, and a pixel-level two-dimensional hazard area segmentation mask is output. In actual use of the deep semantic segmentation network, the acquired visible light image of the earth-rock dam and the final associated feature map obtained after a series of processing steps are merged together in a certain way along the channel dimension as network input data. After processing, the network outputs a two-dimensional image, where the value of each pixel represents whether the pixel location belongs to a hazard area. This forms a mask for segmenting hazard areas on the surface of the earth-rock dam, visually displaying the location of hazard areas such as cracks, seepage, and piping inlets.

[0079] like Figure 2 As shown, this paper proposes a knowledge encoding submodule by constructing a differentiable seepage simulation model, driving the forward differentiable seepage field simulation with the intermediate prediction results of a deep semantic segmentation network, comparing the simulation output with the loss term generated by the associated feature map to optimize the deep semantic segmentation network, and including: The simulation model building unit is used to build a differentiable seepage simulation model, which is based on the parameterized dam geometry and the parameterized material permeability coefficient distribution field. The differentiable physics forward simulation unit is used to take the intermediate prediction results of the deep semantic segmentation network during the training process as internal defects or boundary condition parameters, input them into the differentiable seepage simulation model, perform a forward differentiable seepage field numerical simulation, and output the corresponding pore water pressure distribution cloud map and seepage velocity vector field. The physical field feature encoding unit is used to convert the pore water pressure distribution cloud map and seepage velocity vector field into physical field feature vectors through a differentiable encoding network; The Physical Consistency Loss Generation Unit is used to compare the physical field feature vector with the global feature vector extracted from the final association feature map corresponding to the same sample, which is output by the cross-modal association analysis module. The difference between the two in the feature space is calculated as a dynamic physical constraint loss term to optimize the deep semantic segmentation network.

[0080] In this embodiment, constructing a differentiable seepage simulation model means establishing a model capable of simulating seepage within an earth-rock dam, and this model possesses the characteristic of being differentiable. Differentiability means that during model operation, its internal computational steps can calculate the corresponding gradients in response to changes in input parameters. This allows the model to be combined with the training process of a neural network, optimizing relevant parameters through gradient backpropagation. The construction process is as follows: First, based on the earth-rock dam design drawings and geological survey data, the dam's geometric structure and the distribution of material permeability coefficients are determined and parameterized. For example, geometric dimensions such as dam height and slope, as well as the permeability coefficients of different materials, are represented by variables.

[0081] Based on Darcy's law and other seepage control equations, a numerical model is constructed using discretization methods such as the finite element method or finite volume method. During the construction process, automatic differentiation technology is employed to allow each calculation step within the model to track changes in input parameters and calculate gradients. In this way, a differentiable seepage simulation model is built, enabling it to be combined with neural networks to optimize parameters.

[0082] In this embodiment, the differentiable seepage simulation model is based on parameterized dam geometry and parameterized material permeability coefficient distribution field. These parameterized dam geometry and material permeability coefficient distribution field provide the basic input conditions for the model. Parameterization allows these structures and distribution fields to be adapted to different actual conditions of earth-rock dams by adjusting parameters, thereby enhancing the model's versatility and adaptability.

[0083] In this embodiment, the parameterized dam geometry refers to using a set of parameters to describe the dam's geometry and dimensions. For example, parameters can represent information such as dam height, dam crest width, and upstream and downstream slope gradients.

[0084] In this embodiment, the parameterized material permeability coefficient distribution field refers to representing the material permeability coefficient of different parts of the dam body using parameters.

[0085] In this embodiment, the intermediate prediction results of the deep semantic segmentation network during the training process are the prediction results of the potential hazard areas on the surface of the earth-rock dam at a certain intermediate stage after the deep semantic segmentation network processes the input data during training. Assuming the deep semantic segmentation network is training to analyze an image of an earth-rock dam, when half of the training steps are completed, the network outputs an image. In this image, potential hazard areas such as cracks and seepage areas are initially marked with different colors. This is the prediction result of the potential hazard areas on the surface of the earth-rock dam at a certain intermediate stage.

[0086] In this embodiment, internal defect or boundary condition parameters refer to parameters related to potential defects (such as cracks and pores) and seepage boundary conditions (such as upstream and downstream water levels and seepage outlet conditions) within the earth-rock dam. For example, crack parameters include crack length, width, and location; pore parameters include pore diameter, depth, and location. Upstream and downstream water level parameters refer to the current upstream and downstream water levels, and seepage outlet condition parameters include outlet flow velocity and flow rate. These are parameters related to defects and seepage boundary conditions. The intermediate prediction results of the deep semantic segmentation network can be transformed into these parameters and input into a differentiable seepage simulation model to simulate seepage under these potential hazards.

[0087] In this embodiment, the intermediate prediction results of the deep semantic segmentation network during training are used as parameters for internal defects or boundary conditions. These parameters are then input into a differentiable seepage simulation model to perform a forward differentiable seepage field numerical simulation. The resulting output includes a pore water pressure distribution cloud map and a seepage velocity vector field. The intermediate prediction results of the deep semantic segmentation network are transformed into parameters describing internal defects or boundary conditions and then input into the differentiable seepage simulation model. Based on these parameters and existing information on the dam's geometry and material permeability coefficient distribution, the model simulates the seepage process using numerical calculation methods, obtaining a numerical simulation result of the seepage field, presented in the form of a pore water pressure distribution cloud map and a seepage velocity vector field.

[0088] In this embodiment, the pore water pressure distribution cloud map visually displays the distribution of pore water pressure at various locations within the earth-rock dam. Different colors or shades of gray in the cloud map can represent different ranges of pore water pressure values.

[0089] In this embodiment, the seepage velocity vector field is a way to describe the seepage velocity within an earth-rock dam. In this field, each point has a vector, the length of which represents the magnitude of the seepage velocity at that point, and the direction of the vector represents the direction of seepage.

[0090] In this embodiment, a differentiable coding network is a special network structure that can encode input data and possesses differentiable characteristics during the encoding process. This differentiability allows it to seamlessly connect with other differentiable models or network components (such as differentiable percolation simulation models and deep semantic segmentation networks) for joint optimization through gradient backpropagation. Differentiable coding networks are typically built based on multilayer perceptrons (MLPs) or convolutional neural networks (CNNs). Taking a CNN-based network as an example, the network consists of multiple convolutional layers, pooling layers, and fully connected layers.

[0091] Convolutional layers perform convolution operations by sliding convolution kernels across the input data (such as pore water pressure distribution cloud maps and seepage velocity vector field data) to extract local features, with the parameters of each convolution kernel participating in the calculation. Pooling layers downsample the output of the convolutional layers, reducing data dimensionality while retaining key features. Fully connected layers fully connect the data processed by convolution and pooling, outputting encoded feature vectors.

[0092] Throughout the process, the network is based on automatic differentiation technology and uses the chain rule to calculate the gradient of each operation with respect to the input data, ensuring that the encoding process is differentiable, thereby achieving joint optimization with other differentiable models or network components through gradient backpropagation.

[0093] In this embodiment, a differentiable coding network is used to convert the pore water pressure distribution cloud map and the seepage velocity vector field into a physical field feature vector. The differentiable coding network processes the pore water pressure distribution cloud map and the seepage velocity vector field, transforming these continuous physical field data into a single feature vector. Assume the differentiable coding network consists of a simple three-layer convolutional layer and a fully connected layer. For the pore water pressure distribution cloud map, it first passes through the first convolutional layer, where a specific convolutional kernel extracts local pressure change features to obtain a feature map. Then, it passes through the second convolutional layer to further extract more complex features, and the third convolutional layer further refines the data. The seepage velocity vector field is processed in the same way. Then, both data are dimensionality-reduced through a pooling layer, concatenated, and fed into the fully connected layer. The fully connected layer integrates the concatenated data and outputs a one-dimensional physical field feature vector, completing the transformation from continuous physical field data to a feature vector.

[0094] This feature vector extracts key information from the physical field, enabling subsequent comparison and analysis with other feature vectors. It also leverages the differentiability of the encoding network to support joint optimization of the entire system.

[0095] In this embodiment, the global feature vector extracted from the final correlation feature map corresponding to the same sample, output by the cross-modal correlation analysis module, is a vector that represents the overall characteristics of the sample. This global feature vector integrates information contained in multiple data modes (such as visible light images, infrared thermal imaging, apparent resistivity, distributed fiber optic strain data, etc.), reflecting the correlation between internal hidden dangers and surface features of earth-rock dams.

[0096] In this embodiment, the physical field feature vector is compared with the global feature vector extracted from the final association feature map corresponding to the same sample, output by the cross-modal association analysis module. A difference metric is calculated between the two in the feature space. The physical field feature vector obtained through the differentiable coding network and the global feature vector extracted from the final association feature map are compared in the same feature space. The degree of difference between them is calculated using an Euclidean distance algorithm or a pre-similarity algorithm. This difference metric reflects the inconsistency between the physical field simulated based on the intermediate prediction results of the deep semantic segmentation network and the actual situation reflected by the cross-modal association analysis.

[0097] In this embodiment, a dynamic physical constraint loss term is used to optimize the deep semantic segmentation network. The difference metric calculated above is used as the physical constraint loss term. Since this loss term is dynamically generated based on the intermediate prediction results of each deep semantic segmentation network, it can reflect the deviation between the network's prediction results and the actual physical situation in real time. This loss term is incorporated into the training loss function of the deep semantic segmentation network, and the network parameters are adjusted through gradient backpropagation. This allows the network to more accurately predict the potential hazard areas on the surface of earth-rock dams that conform to the actual seepage physical laws during subsequent training, thereby optimizing the performance of the deep semantic segmentation network.

[0098] like Figure 2 As shown, by constructing a physical information neural network training framework, combining image input, seepage field simulation and multiple loss calculations within a single iteration cycle, and synchronously optimizing the parameters of the deep semantic segmentation network and the differentiable seepage simulation model through gradient backpropagation, the accuracy of hazard identification is improved. A differentiable seepage simulation model, a differentiable physical forward simulation unit and a network training and application sub-module in the physical constraint image recognition module are proposed to jointly constitute a physical information neural network training framework. The training process of the physical information neural network training framework executes the following steps sequentially within a single iteration cycle: The visible light image and the final associated feature map are input into a deep semantic segmentation network to perform image feature encoding and initial hazard area prediction. The initial hazard area prediction is mapped to the internal defect parameters of the differentiable seepage simulation model, which drives the differentiable seepage simulation model to perform seepage field simulation. The physical consistency of the physical quantity distribution of the seepage field output by the simulation is compared with the final associated feature map. The total training loss of the deep semantic segmentation network consists of pixel-level classification loss, physical constraint loss generated based on physical consistency comparison, and field reconstruction loss. The field reconstruction loss is used to measure the direct difference between the final associated feature map and the distribution of physical quantities of the seepage field output by the differentiable seepage simulation model. By using gradient backpropagation, the weight parameters of the deep semantic segmentation network and the material permeability coefficient parameters in the differentiable seepage simulation model are simultaneously optimized.

[0099] In this embodiment, the differentiable seepage simulation model, the differentiable physical forward simulation unit, and the network training and application sub-module in the physical constraint image recognition module together constitute a physical information neural network training framework. This means combining the differentiable seepage simulation model that can simulate seepage and calculate gradients, the differentiable physical forward simulation unit responsible for inputting intermediate prediction results from the deep semantic segmentation network into the model and executing the simulation, and the network training and application sub-module that constructs and trains the deep semantic segmentation network and applies it for hazard identification, into a unified framework. This framework combines seepage knowledge from the physical domain with neural networks to train the deep semantic segmentation network, enabling it to better identify hazards in earth-rock dams.

[0100] In this embodiment, the physical information neural network training framework is a system that integrates a physical model and a neural network training process. It utilizes a differentiable seepage simulation model to simulate physical processes, combines the image processing capabilities of a deep semantic segmentation network, and optimizes network parameters and physical model parameters through a series of steps and loss function calculations to improve the accuracy and physical realism of identifying potential hazards in earth-rock embankments.

[0101] In this embodiment, the visible light image and the final associated feature map are input into a deep semantic segmentation network for image feature encoding and initial hazard area prediction. Specifically, the acquired visible light image of the earth-rock embankment surface, along with the final associated feature map reflecting the probability of internal hazards output by the cross-modal association analysis module, are input into the deep semantic segmentation network. The network processes the image, extracts and encodes its feature information, and then uses this encoded information to initially predict areas on the earth-rock embankment surface that may contain hazards.

[0102] In this embodiment, the initial predicted potential hazard area is mapped to the internal defect parameters of a differentiable seepage simulation model. This drives the differentiable seepage simulation model to perform seepage field simulation, and the distribution of physical quantities in the seepage field output by the simulation is compared with the final associated feature map for physical consistency. Specifically, the potential hazard area initially predicted by the deep semantic segmentation network is transformed into parameters representing internal defects in the differentiable seepage simulation model, such as the location and size of cracks. Using these parameters, the differentiable seepage simulation model is driven to simulate the seepage field within the earth-rock dam. After the model runs, it outputs the distribution of seepage field-related physical quantities, such as pore water pressure and seepage velocity. These distributions are compared with the final associated feature map to see if they are physically consistent; for example, whether the high-pressure areas in the seepage field match the potential hazard areas implied in the final associated feature map.

[0103] In this embodiment, the internal defect parameters of the differentiable seepage simulation model are derived from the initial hazard region prediction of the deep semantic segmentation network and are used to describe the possible defects inside the earth-rock dam. These parameters affect the operation of the differentiable seepage simulation model, enabling the model to simulate the seepage field when these defects are present. For example, parameters such as the shape, location, and size of the defects will change the seepage path and velocity.

[0104] In this embodiment, the seepage field simulation is a differentiable seepage simulation model that uses input information such as internal defect parameters, dam geometry parameters, and material permeability coefficient distribution to simulate the actual seepage process inside an earth-rock dam through numerical calculation. Under given conditions, it calculates seepage-related physical quantities at various locations within the dam, such as pore water pressure and seepage velocity, thus obtaining the distribution of the seepage field.

[0105] Suppose we simplify an earth-rock dam into a regular rectangular grid. Known dam geometry parameters determine the grid shape and size, the material permeability distribution field gives the permeability of each grid cell, and internal defect parameters include, for example, introducing a crack into a certain grid cell.

[0106] The finite difference method, based on Darcy's law, is employed. Taking the calculation of pore water pressure as an example, an initial pressure value is set for each grid node. For adjacent nodes, the hydraulic gradient is calculated based on the permeability coefficient and the distance between nodes. According to Darcy's law, the seepage velocity between nodes is calculated from the hydraulic gradient, and the pore water pressure value is then updated. This process is repeated to gradually obtain the pore water pressure at each location in the entire dam grid. The seepage velocity is calculated using a similar method, ultimately obtaining the seepage field distribution.

[0107] In this embodiment, the distribution of physical quantities in the seepage field output by the simulation is the result obtained after calculation by the differentiable seepage simulation model, showing the specific numerical distribution of seepage-related physical quantities at various locations inside the earth-rock dam. For example, the pore water pressure distribution cloud map shows the distribution of pore water pressure at various locations in the dam body, and the seepage velocity vector field shows the distribution of the magnitude and direction of seepage velocity at various locations in the dam body.

[0108] In this embodiment, the total training loss of the deep semantic segmentation network consists of pixel-level classification loss, physical constraint loss generated based on physical consistency comparison, and field reconstruction loss. The total training loss is a metric for evaluating the training performance of the deep semantic segmentation network, and it comprises three parts. These three parts measure the difference between the network's predictions and the actual situation from different perspectives, working together to make the network more accurate and more consistent with physical reality.

[0109] In this embodiment, pixel-level classification loss is used to measure the accuracy of the deep semantic segmentation network in classifying each pixel in the image. It calculates the difference between the network's prediction that each pixel belongs to a hazardous area or not and the actual situation, prompting the network to more accurately identify the category of each pixel and improve the accuracy of hazardous area segmentation.

[0110] In this embodiment, the physical constraint loss generated based on the physical consistency comparison is calculated according to the physical consistency comparison results between the distribution of physical quantities of the seepage field output by the differentiable seepage simulation model and the final associated feature map. If the difference between the two is large, it indicates that the hidden danger area predicted by the network does not match the actual physical situation, and this loss will increase, thus constraining the prediction results of the network to be more in line with physical laws.

[0111] In this embodiment, the field reconstruction loss is used to measure the direct difference between the final associated feature map and the distribution of physical quantities of the seepage field output by the differentiable seepage simulation model. It prompts the deep semantic segmentation network to not only consider pixel classification and physical consistency, but also to ensure that the prediction results can better reconstruct features similar to the actual distribution of physical quantities of the seepage field, further improving the fit between the network prediction and the actual physical situation. Here, the direct difference refers to the degree of inconsistency between the information reflected in the final associated feature map and the information reflected in the distribution of physical quantities of the seepage field. For example, comparing the correspondence between high-value and low-value regions in spatial location, as well as the differences in numerical magnitude, etc., the network parameters are adjusted through the quantification of this difference.

[0112] In this embodiment, the distribution of physical quantities of the seepage field output by the differentiable seepage simulation model is the result obtained after the model simulates the seepage process, including the distribution of physical quantities such as pore water pressure and seepage velocity at various locations inside the earth-rock dam. These distribution results reflect the actual state of the seepage field under given internal defect parameters and other conditions.

[0113] In this embodiment, gradient backpropagation is used to simultaneously optimize the parameters of the deep semantic segmentation network and the differentiable seepage simulation model.

[0114] In this embodiment, gradient backpropagation is used to simultaneously optimize the weight parameters of the deep semantic segmentation network and the material permeability coefficient parameters in the differentiable seepage simulation model. During training, the gradients with respect to the weight parameters of the deep semantic segmentation network and the material permeability coefficient parameters of the differentiable seepage simulation model are calculated based on the total training loss. Then, along the opposite direction of the gradient, the weight parameters of the deep semantic segmentation network are adjusted simultaneously to allow the network to better identify potential hazard areas, and the material permeability coefficient parameters in the differentiable seepage simulation model are adjusted to make the model more accurately simulate actual seepage conditions.

[0115] In this embodiment, the weight parameters of the deep semantic segmentation network determine what transformations and processes the network performs on the input data. These parameters are continuously adjusted during training, and through gradient backpropagation and feedback from the loss function, the network can better extract image features and accurately segment the potential hazard areas on the surface of the earth-rock embankment.

[0116] In this embodiment, the material permeability coefficient parameter in the differentiable seepage simulation model describes the permeability of different materials in the dam body. Different materials have different permeability coefficients, and these parameters affect the flow velocity and path of seepage within the dam body. During training, these parameters are optimized through gradient backpropagation to make the model simulate seepage conditions more consistent with the actual seepage state within an earth-rock dam.

[0117] To construct a 3D solid model of a hazard by back-projecting a 2D hazard segmentation mask onto a 3D spatial coordinate system and binding it with time information, and then to assess the risk and generate early warning information based on historical models and real-time operating parameters, thereby achieving 3D visualization and dynamic risk early warning of hazards, a hazard 3D visualization and early warning module is proposed, including: The 3D modeling and mapping submodule is used to segment the 2D hidden danger area mask, and based on the spatial correspondence with the lidar point cloud, project it back into the 3D spatial coordinate system to construct a 3D solid model of the earth-rock dam hidden danger with geometric dimensions and spatial location, and bind it with temporal information. The dynamic risk assessment submodule is used to analyze the expansion speed and morphological change trend of hidden dangers based on the historical multi-period three-dimensional entity model sequence. Combined with real-time reservoir water level and rainfall operation parameters, the risk assessment model is used to calculate the risk level of the current hidden danger and the probability of its future evolution, and generate graded early warning information.

[0118] In this embodiment, a two-dimensional hazard area segmentation mask is projected back into a three-dimensional spatial coordinate system based on its spatial correspondence with the LiDAR point cloud. This constructs a three-dimensional solid model of the earth-rock embankment hazard with geometric dimensions and spatial location, and binds it with temporal information. In other words, the previously obtained two-dimensional hazard area segmentation mask displays the hazard area on the surface of the earth-rock embankment. Since the LiDAR point cloud data constructs a three-dimensional spatial coordinate system and has a certain spatial correspondence with the two-dimensional image, this correspondence is used to project the hazard area on the two-dimensional mask back into the three-dimensional spatial coordinate system. This allows the construction of a three-dimensional solid model of the earth-rock embankment hazard that not only includes the geometric dimensions of the hazard area but also clearly defines its location in three-dimensional space. Furthermore, time-series information is bound to this model to record the time of hazard occurrence and change.

[0119] In this embodiment, the spatial correspondence with the lidar point cloud refers to the correspondence between each position in the two-dimensional hidden danger area segmentation mask and the position in the three-dimensional spatial coordinate system constructed by the lidar point cloud.

[0120] In this embodiment, the historical multi-period hidden danger 3D entity model sequence is a sequence formed by arranging multiple 3D entity models of earth-rock dam hidden dangers acquired and constructed at different times in chronological order. Each model records the hidden danger status at the corresponding time point.

[0121] In this embodiment, the expansion rate and morphological change trend of potential hazards are analyzed based on a historical sequence of three-dimensional solid models of potential hazards from multiple periods. This involves comparing the geometric dimensions, shape, and location of hazard areas in models from different periods. By calculating the changes in hazard areas between adjacent periods and combining this with the time interval, the expansion rate of the hazard is determined. Observing the changes in the shape of hazard areas at different times allows for the summarization of morphological change trends, providing insight into how hazards develop over time.

[0122] In this embodiment, by combining real-time reservoir water level and rainfall parameters, a risk assessment model is used to calculate the current risk level and future evolution probability of a potential hazard, generating tiered early warning information. The real-time reservoir water level refers to the actual height of the water level in the reservoir at the current moment. Rainfall parameters include the total rainfall, rainfall intensity, and rainfall temporal distribution over a period of time. The risk assessment model is a pre-established mathematical model that comprehensively considers factors such as the real-time reservoir water level, rainfall parameters, and the expansion rate and morphological change trend of the potential hazard. Through the calculations of this model, the current risk level of the potential hazard is determined, i.e., the degree to which the hazard threatens the safety of the earth-rock dam, and the probability of future evolution of the hazard, such as the possibility of further expansion and the occurrence of more serious problems. Based on these calculation results, different levels of early warning information are generated to remind relevant personnel of different levels of risk.

[0123] In this embodiment, the risk assessment model is a model constructed using mathematical methods and algorithms based on relevant data and hazard characteristics of earth-rock dams. It integrates information from multiple aspects, including real-time reservoir water level, rainfall parameters, hazard expansion rate, and morphological change trends, to assess the current risk level of the hazard and the likelihood of future changes, providing a basis for generating early warning information. The information regarding its construction process is as follows: Construction method: Collect a large amount of historical data on earth and rock dams, including real-time reservoir water level, rainfall, hazard expansion and morphological changes, as well as corresponding actual risk assessment results.

[0124] Algorithm used: Logistic regression can be selected. It can handle the complex relationships between multiple factors and risk levels, and find the correlation patterns between various factors and risk levels by learning from historical data.

[0125] Training set inputs include: real-time reservoir water level values, rainfall-related parameters (such as total amount and intensity), data on the rate of hazard expansion, and data describing morphological change trends.

[0126] Training set output: The actual level of potential risks under each combination of input data, such as low, medium, and high risk levels, and the probability of future evolution.

[0127] In this embodiment, the current risk level of the hidden danger and the probability of its future evolution are considered. The risk level is a quantitative assessment of the severity of the current hidden danger in the earth-rock dam, such as being divided into low, medium, and high risk levels, indicating the magnitude of the threat the hidden danger poses to the dam's safety. The probability of future evolution is calculated based on existing data and models, representing the likelihood that the hidden danger will undergo specific changes in the future, such as the probability of the hidden danger expanding or connecting to form a more serious problem.

[0128] In this embodiment, the tiered early warning information is generated based on the current risk level of potential hazards and the probability of their future evolution, calculated by a risk assessment model, resulting in different levels of warning messages. For example, a general warning may be issued for low-risk levels to remind people to pay attention; while an emergency alarm may be issued for high-risk levels, requiring immediate action so that relevant personnel can respond appropriately according to different levels of risk to ensure the safety of earth-rock dams.

[0129] like Figure 2 As shown, to generate initial boundary conditions by combining the current three-dimensional distribution model of potential hazards with extreme working conditions, and to iteratively deduce the future evolution of potential hazards through a physical information neural network training framework, statistical analysis of simulation results generates graded forward-looking early warning signals to accurately assess the future risks of potential hazards, a dynamic risk assessment submodule is proposed, including: The multi-scenario boundary initialization unit is used to generate a set of initial boundary conditions for multiple future scenarios, based on the three-dimensional distribution model of potential hazards in earth-rock dams identified at the current moment, combined with preset extreme working condition reservoir water level curves and rainfall process lines. The physical information cyclic deduction unit is used to input the corresponding set of initial boundary conditions into the physical information neural network training framework whose parameters have been optimized and converged for each deduction scenario. In the deduction mode, the weights of the fixed deep semantic segmentation network and the material parameters of the differentiable seepage simulation model are used as the input conditions for the next time step, along with the hazard segmentation results and seepage field state output by the framework of the previous time step. The coupling evolution process of the hazard in multiple consecutive time steps is simulated in an autoregressive cyclic iteration manner. The risk quantification unit is used to statistically analyze the simulation results of all cyclical scenarios, calculate the first occurrence time and cumulative probability of the hidden danger expanding to the critical geometric size or the local seepage gradient exceeding the soil's allowable seepage resistance strength, and generate graded early warning signals based on the corresponding probability timeline.

[0130] In this embodiment, the three-dimensional distribution model of hidden dangers in the earth-rock embankment identified at the current moment refers to the model constructed at this point in time by detecting and analyzing the earth-rock embankment and using the series of methods mentioned above (such as multimodal data acquisition, processing and analysis). This model can show the distribution of hidden dangers inside and on the surface of the earth-rock embankment in three-dimensional space.

[0131] In this embodiment, the preset extreme condition reservoir water level curve and rainfall process curve are pre-defined curves that simulate reservoir water level changes and rainfall under extreme conditions. The extreme condition reservoir water level curve describes the trend of reservoir water level changes over time under extreme conditions such as floods, including information such as the magnitude and rate of rise and fall. The rainfall process curve represents the distribution of rainfall over time under extreme rainfall conditions, such as the intensity of the rainstorm, the duration of rainfall, and the temporal variation of rainfall. These curves are used to simulate the impact of potential severe conditions on earth-rock dams.

[0132] In this embodiment, the core is the three-dimensional distribution model of potential hazards in the earth-rock dam identified at the current moment. Combined with preset extreme reservoir water level curves and rainfall process lines, a set of initial boundary conditions for multiple future scenarios is generated. Essentially, based on the currently constructed three-dimensional hazard distribution model, and considering preset extreme reservoir water levels and rainfall conditions, a set of initial conditions is set for each possible future scenario. For example, based on different extreme reservoir water level starting values, rainfall start times and intensities, and combined with the current hazard distribution, the initial state of the earth-rock dam under each scenario is determined. The set of these initial states constitutes the set of initial boundary conditions for the future scenarios.

[0133] In this embodiment, the future simulation scenarios are simulations of different possible situations that may occur in the earth-rock dam in the future. Each scenario is based on the current distribution of hidden dangers and preset extreme working conditions, taking into account the impact of factors such as reservoir water level and rainfall on the development of hidden dangers, and simulating the possible evolution process and results of hidden dangers under different conditions.

[0134] In this embodiment, the set of initial boundary conditions for future scenarios is a summary of the starting conditions set for each future scenario. These conditions include the initial hazard state of the earth-rock dam (from the current hazard three-dimensional distribution model), as well as initial conditions related to extreme conditions, such as the initial reservoir water level and initial rainfall. These conditions provide a starting point for subsequent simulations of each scenario.

[0135] In this embodiment, the optimized and converged physical information neural network training framework is a framework that has undergone a series of training processes. The weight parameters of the deep semantic segmentation network and the material parameters of the differentiable seepage simulation model have reached a stable state and no longer show significant changes. This framework enables the model to achieve good results in identifying and simulating potential hazards in earth-rock dams. This framework combines physical knowledge and neural networks for subsequent deduction of hazard development.

[0136] In this embodiment, under the simulation mode, the weights of the deep semantic segmentation network and the material parameters of the differentiable seepage simulation model are fixed. The hazard segmentation results and seepage field state output by the previous time step are used as the input conditions for the next time step. An autoregressive iterative process is used to simulate the coupled evolution of the hazard over multiple consecutive time steps. This means that when simulating the future evolution of the hazard, the weights of the deep semantic segmentation network and the material parameters of the differentiable seepage simulation model are not adjusted. The segmentation results of the hazard region and the state information of the seepage field output by the previous time step are used as the input data for the simulation of the next time step. This process is repeated continuously in an autoregressive iterative manner to simulate the interaction and co-change between the hazard and the seepage field over multiple consecutive time steps, thereby observing the future development of the hazard.

[0137] In this embodiment, the weights of the deep semantic segmentation network and the material parameters of the differentiable seepage simulation model are used. The weights of the deep semantic segmentation network determine how the network extracts features and classifies the input data, and are continuously adjusted during training to optimize network performance. The material parameters of the differentiable seepage simulation model describe the permeability characteristics of different materials in the dam body, such as the permeability coefficient, which affects the flow of seepage within the dam body. In the simulation mode, these two sets of parameters are fixed to maintain the stability and consistency of the simulation process.

[0138] In this embodiment, the hazard segmentation results and seepage field state output by the previous time step framework refer to the segmentation of the hazard area of ​​the earth-rock embankment and the distribution of various physical quantities (such as pore water pressure and seepage velocity) in the seepage field obtained by the physical information neural network training framework after the previous time step is completed during the simulation in chronological order. These results reflect the condition of the earth-rock embankment at the previous moment and serve as the basic input for the simulation at the next moment.

[0139] In this embodiment, the input conditions for the next time step are the hazard segmentation results and seepage field state output by the framework of the previous time step, which provide the starting state information for the simulation of the next time step. By continuously using the results of the previous step as the input for the next step, continuous simulation of the changes of hazards and seepage field over time is achieved.

[0140] In this embodiment, the autoregressive iterative approach is a computational method used in the simulation process. It calculates the state of the next time step based on the output of the previous time step. Each iteration depends on the result of the previous iteration, and this process is repeated continuously to gradually simulate the evolution of potential hazards and the seepage field over time. The result of each step is passed to the next step as input, continuously advancing the simulation process.

[0141] In this embodiment, the coupled evolution of the hidden danger over multiple consecutive time steps refers to the process in which the development of the hidden danger (such as expansion and morphological changes) and the changes in the seepage field (such as changes in seepage velocity and direction, and changes in pore water pressure) influence and interact with each other during the simulation. As time progresses, the state of the hidden danger affects the seepage field at each time step, and the changes in the seepage field, in turn, affect the development of the hidden danger. The two are closely related and evolve together.

[0142] In this embodiment, the simulation results of all cyclical scenarios are statistically analyzed to calculate the first occurrence time and cumulative probability of a hazard expanding to a critical geometric size or a local seepage gradient exceeding the soil's allowable impermeability. Based on the corresponding probability timeline, graded early warning signals are generated. This involves organizing and analyzing the results of all simulated future scenarios. The calculations determine the first occurrence time and cumulative probability of these situations—when the hazard expands to a specific geometric size that could lead to serious problems, or when the local seepage gradient exceeds the soil's impermeability. Then, based on the distribution of these probabilities over time (i.e., the probability timeline), different levels of early warning signals are generated to promptly alert relevant personnel to varying degrees of risk.

[0143] In this embodiment, the simulation results for all cyclical scenarios refer to the detailed data and descriptions of the hazard development and seepage field changes obtained after simulating each future scenario. This includes information such as the hazard status at each time step and the values ​​of various physical quantities in the seepage field.

[0144] In this embodiment, the critical geometric dimension refers to the size at which potential hazards in an earth-rock dam may lead to serious safety problems, such as damage to the dam structure or increased leakage.

[0145] In this embodiment, the local seepage gradient exceeds the soil's allowable impermeability strength. The seepage gradient refers to the head loss per unit seepage length. When the local seepage gradient exceeds the soil's ability to resist seepage damage (i.e., the soil's allowable impermeability strength), the soil may undergo seepage deformation, such as piping and soil erosion, which seriously threatens the safety of the earth-rock dam.

[0146] In this embodiment, the cumulative probability refers to the total probability that, across all cyclic simulation scenarios, the hazard will expand to a critical geometric size or the local seepage gradient will exceed the soil's allowable impermeability. It comprehensively considers the likelihood of these dangerous situations occurring under different scenarios, reflecting the overall level of risk.

[0147] In this embodiment, the probability timeline is a curve with time on the horizontal axis and the cumulative probability of a hazard expanding to a critical geometric size or a local seepage gradient exceeding the soil's allowable impermeability strength on the vertical axis. It demonstrates the changing trends in the probability of these hazardous situations occurring over time, providing an intuitive basis for generating graded early warning signals.

[0148] In this embodiment, the tiered early warning signal generates different levels of alert information based on the probability timeline and the varying degrees of development of potential hazards. For example, when the cumulative probability is low, a lower-level warning is issued to alert the user; when the cumulative probability is high, approaching a level where danger may occur, a higher-level warning is issued to prompt emergency measures to ensure the safety of the earth-rock embankment.

[0149] This invention provides an embodiment of a cloud platform for analyzing potential hazards in earth-rock dams based on image processing, comprising: Memory, used to store computer programs; A processor connected to memory is used to execute computer programs stored in memory to implement any of the above systems.

[0150] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of this invention and its equivalents, this invention also intends to include these modifications and variations.

Claims

1. A system for analyzing potential hazards in earth-rock embankments based on image processing, characterized in that, include: The multimodal data acquisition module is used to simultaneously acquire visible light images, infrared thermal imaging data, lidar point cloud data of the surface of the earth-rock embankment, as well as apparent resistivity data and distributed fiber strain data reflecting the internal state of the earth-rock embankment. The data registration and fusion module is used to establish a unified three-dimensional spatial coordinate system based on lidar point cloud data, and to register visible light images, infrared thermal imaging data, apparent resistivity data and distributed fiber strain data to the three-dimensional spatial coordinate system to form a spatially aligned multi-source data field. The cross-modal correlation analysis module is used to construct a graph neural network model. The graph neural network model takes a spatially aligned multi-source data field as input and learns the nonlinear correlation between surface texture features in visible light images, temperature anomaly features in infrared thermal imaging data, resistivity anomaly features in apparent resistivity data, and strain abrupt change features in distributed optical fiber strain data. It outputs the final correlation feature map that characterizes the probability of internal hidden dangers. The physical constraint image recognition module is used to encode the prior knowledge of seepage mechanics of earth-rock dams into constraint conditions and construct a deep semantic segmentation network that integrates the constraint conditions. The deep semantic segmentation network takes visible light images and associated feature maps as joint inputs, performs pixel-level classification, identifies and segments the cracks, seepage, and piping inlet hazard areas on the surface of earth-rock dams, and uses them as a two-dimensional hazard area segmentation mask. The 3D visualization and early warning module for hidden dangers is used to map the 2D hidden danger area segmentation mask back to the 3D spatial coordinate system, generate a 3D distribution model of hidden dangers in earth and rock embankments that integrates spatial location and monitoring time information, and trigger graded early warning signals based on the evolution trend of hidden dangers.

2. The image processing-based hazard analysis system for earth-rock embankments according to claim 1, characterized in that, The data registration and fusion module includes: The coordinate system submodule is used to establish a three-dimensional spatial coordinate system with real geographic coordinates or engineering coordinates, using the high-precision three-dimensional point cloud obtained by lidar scanning as a spatial reference. The multi-source registration and alignment submodule is used to match feature points in visible light images and infrared thermal imaging with lidar point clouds through feature extraction and matching algorithms, and to map the measurement point positions of apparent resistivity data and distributed fiber strain data to a three-dimensional spatial coordinate system through spatial interpolation algorithms to obtain a spatially aligned multi-source data field.

3. The image processing-based hazard analysis system for earth-rock dams according to claim 1, characterized in that, The cross-modal correlation analysis module includes: The heterogeneous graph construction submodule is used to construct a heterogeneous graph data structure based on spatially aligned multi-source data fields. The nodes of the heterogeneous graph include image pixel nodes, resistivity measurement point nodes, and fiber strain measurement point nodes. The node feature vector contains the feature values ​​extracted from the corresponding data source. The edges of the heterogeneous graph are connected based on the similarity between the Euclidean distance of the nodes in the three-dimensional spatial coordinate system and the node feature vector, and are assigned initial weights. The associated feature learning submodule is used to input the constructed heterogeneous graph data structure into the graph neural network model. Through a multi-layer message passing mechanism, the node features are iteratively updated and aggregated, so that the final feature vector of each node is fused with the association information of cross-modal nodes in the corresponding neighborhood, and the final associated feature map representing the probability of internal hidden dangers is output.

4. The image processing-based hazard analysis system for earth-rock dams according to claim 1, characterized in that, The associated feature learning submodule includes: The associated feature learning unit is used to input the constructed heterogeneous graph data structure into the graph neural network model. Through a multi-layer message passing mechanism, iterative updates and aggregation of node features are performed, so that the final feature vector of each node is fused with the association information of cross-modal nodes in the corresponding neighborhood, and the initial associated feature map representing the probability of internal hidden dangers is output. The confidence filtering unit is used to calculate the feature confidence value of each pixel position in the initial associated feature map and filter out isolated noise responses with confidence values ​​lower than a preset threshold to obtain the initial associated feature map after confidence filtering. The region fusion enhancement unit is used to fuse and smooth the discrete high-response regions based on spatial proximity and feature similarity in the initial association feature map after confidence filtering, generating a final association feature map with continuous spatial distribution and corresponding regional connectivity that matches the scale of typical physical anomaly regions inside the earth-rock dam.

5. The image processing-based hazard analysis system for earth-rock dams according to claim 1, characterized in that, The physical constraint image recognition module includes: The knowledge encoding submodule is used to encode the prior knowledge of seepage mechanics of earth-rock dams into computable constraint rules, and generate physical constraint loss terms for training deep semantic segmentation networks based on the constraint rules. The prior knowledge of seepage mechanics of earth-rock dams includes constraints on the distribution range of the seepage line, constraints on the directionality of the seepage path, and constraints on the synergy between surface hazards and internal abnormal signals. The network training and application submodule is used to build and train a deep semantic segmentation network. The training loss function of the deep semantic segmentation network consists of pixel-level classification loss, region structure loss, and physical constraint loss. When the network is applied, the visible light image and the final associated feature map are concatenated as network input, and a pixel-level two-dimensional hazard region segmentation mask is output.

6. The image processing-based hazard analysis system for earth-rock dams according to claim 1, characterized in that, The knowledge encoding submodule includes: The simulation model building unit is used to build a differentiable seepage simulation model, which is based on the parameterized dam geometry and the parameterized material permeability coefficient distribution field. The differentiable physics forward simulation unit is used to take the intermediate prediction results of the deep semantic segmentation network during the training process as internal defects or boundary condition parameters, input them into the differentiable seepage simulation model, perform a forward differentiable seepage field numerical simulation, and output the corresponding pore water pressure distribution cloud map and seepage velocity vector field. The physical field feature encoding unit is used to convert the pore water pressure distribution cloud map and seepage velocity vector field into physical field feature vectors through a differentiable encoding network; The Physical Consistency Loss Generation Unit is used to compare the physical field feature vector with the global feature vector extracted from the final association feature map corresponding to the same sample, which is output by the cross-modal association analysis module. The difference between the two in the feature space is calculated as a dynamic physical constraint loss term to optimize the deep semantic segmentation network.

7. The image processing-based hazard analysis system for earth-rock dams according to claim 6, characterized in that, The differentiable seepage simulation model, the differentiable physical forward simulation unit, and the network training and application sub-module in the physical constraint image recognition module together constitute a physical information neural network training framework. The training process of the physical information neural network training framework executes the following steps sequentially within a single iteration cycle: The visible light image and the final associated feature map are input into a deep semantic segmentation network to perform image feature encoding and initial hazard area prediction. The initial hazard area prediction is mapped to the internal defect parameters of the differentiable seepage simulation model, which drives the differentiable seepage simulation model to perform seepage field simulation. The physical consistency of the physical quantity distribution of the seepage field output by the simulation is compared with the final associated feature map. The total training loss of the deep semantic segmentation network consists of pixel-level classification loss, physical constraint loss generated based on physical consistency comparison, and field reconstruction loss. The field reconstruction loss is used to measure the direct difference between the final associated feature map and the distribution of physical quantities of the seepage field output by the differentiable seepage simulation model. By using gradient backpropagation, the weight parameters of the deep semantic segmentation network and the material permeability coefficient parameters in the differentiable seepage simulation model are simultaneously optimized.

8. The image processing-based hazard analysis system for earth-rock dams according to claim 1, characterized in that, The hazard 3D visualization and early warning module includes: The 3D modeling and mapping submodule is used to segment the 2D hidden danger area mask, and based on the spatial correspondence with the lidar point cloud, project it back into the 3D spatial coordinate system to construct a 3D solid model of the earth-rock dam hidden danger with geometric dimensions and spatial location, and bind it with temporal information. The dynamic risk assessment submodule is used to analyze the expansion speed and morphological change trend of hidden dangers based on the historical multi-period three-dimensional entity model sequence. Combined with real-time reservoir water level and rainfall operation parameters, the risk assessment model is used to calculate the risk level of the current hidden danger and the probability of its future evolution, and generate graded early warning information.

9. The image processing-based hazard analysis system for earth-rock dams according to claim 8, characterized in that, The dynamic risk assessment submodule includes: The multi-scenario boundary initialization unit is used to generate a set of initial boundary conditions for multiple future scenarios, based on the three-dimensional distribution model of potential hazards in earth-rock dams identified at the current moment, combined with preset extreme working condition reservoir water level curves and rainfall process lines. The physical information cyclic deduction unit is used to input the corresponding set of initial boundary conditions into the physical information neural network training framework whose parameters have been optimized and converged for each deduction scenario. In the deduction mode, the weights of the fixed deep semantic segmentation network and the material parameters of the differentiable seepage simulation model, the hazard segmentation results and seepage field state output by the framework of the previous time step are used as the input conditions for the next time step. The coupling evolution process of the hazard in multiple consecutive time steps in the future is simulated in an autoregressive cyclic iteration manner. The risk quantification unit is used to statistically analyze the simulation results of all cyclical scenarios, calculate the first occurrence time and cumulative probability of the hidden danger expanding to the critical geometric size or the local seepage gradient exceeding the soil's allowable seepage resistance strength, and generate graded early warning signals based on the corresponding probability timeline.

10. A cloud platform for analyzing potential hazards in earth-rock embankments based on image processing, characterized in that, include: Memory, used to store computer programs; A processor connected to a memory for executing a computer program stored in the memory to implement the system according to any one of claims 1-9.

Images