Multi-source data fusion-based water conservancy project defect identification and quantitative evaluation method and system

Abstract

This invention discloses a method and system for defect identification and quantitative assessment in hydraulic engineering projects using multi-source data fusion. The method includes: acquiring multimodal data; extracting spatial topological features from point clouds using a spatiotemporal graph convolutional network; and using these features as a basis, fusing acoustic time-frequency features and visual features across modalities using a multi-head mutual attention mechanism. Then, a Transformer-based sequence extraction module is used to locate and extract defect-sensitive spatiotemporal segments from the fused features. For this segment, a decoder using a fusion physical information neural network is used for 3D reconstruction. During reconstruction, the decoder simultaneously applies physical constraints from observational data and material mechanics equations to generate a physically consistent 3D defect model. Based on this model, defect judgment and quantitative assessment are performed, outputting the identification results, the 3D model, and an assessment report. This method achieves deep fusion of multi-source information and accurate 3D reconstruction guided by physical laws, improving the automation and quantification level of defect detection in hydraulic engineering projects.

Description

Technical Field

[0001] This invention relates to the field of structural health monitoring and non-destructive testing technology for hydraulic engineering projects, specifically to a method and system for intelligent identification and quantitative assessment of structural defects in hydraulic engineering projects based on multi-source data fusion. Background Technology

[0002] Hydraulic engineering projects, such as dams, sluices, water conveyance tunnels, and canal systems, operate for extended periods in complex hydrological, geological, and climatic environments. Their concrete structures are prone to various defects, including cracks, spalling, leakage, internal voids, and steel reinforcement corrosion. Failure to detect and quantitatively assess these defects in a timely manner will seriously threaten the safety and operational lifespan of the project. Traditional manual inspection methods are inefficient, subjective, and struggle to detect hidden defects. Existing automated inspection technologies largely rely on single sensors, resulting in limitations such as limited detection dimensions, insufficient accuracy, and the inability to achieve three-dimensional quantification of defects.

[0003] Currently, defect detection in hydraulic engineering projects mainly employs various non-destructive testing technologies, including visual imaging, infrared thermal imaging, laser scanning, and acoustic testing. Visible light images are convenient for identifying surface cracks and spalling, but cannot detect internal defects; infrared thermal imaging can reflect internal temperature anomalies, but is easily affected by environmental interference and quantitative analysis is difficult; three-dimensional laser scanning can accurately acquire the surface geometry of a structure, but it is difficult to perceive the internal state of the material; acoustic impact echo method is sensitive to defects such as internal voids and looseness, but relies on human experience and has insufficient positioning accuracy. Although some studies have attempted to integrate multiple data for detection, existing methods usually perform simple stitching or weighting at the data level, failing to achieve deep cross-modal feature semantic alignment and complementarity. In addition, existing technologies mostly focus on identifying the presence or absence of defects, and are severely lacking in the ability to quantitatively assess the precise three-dimensional geometry, spatial location, size, and development degree of defects, which is crucial for assessing the severity of defects and making scientific maintenance decisions.

[0004] In recent years, deep learning technology has provided new possibilities for multi-source data fusion and automatic defect identification. However, most current deep learning-based methods are purely data-driven models, and their reconstruction or identification results lack clear physical constraints, which may generate pseudo-defects that do not conform to the principles of materials mechanics, resulting in low reliability of evaluation results and limited engineering guidance value. At the same time, existing models are usually computationally complex and difficult to deploy on mobile or edge computing platforms for on-site inspection of water conservancy projects, limiting their real-time application capabilities.

[0005] Therefore, there is an urgent need to develop an intelligent detection method and system that can deeply integrate multi-source heterogeneous data, accurately identify defects while achieving three-dimensional quantitative reconstruction, and whose reconstruction results conform to engineering physical laws and can adapt to complex field environments. This invention aims to solve the above problems. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method and system for identifying and quantitatively assessing defects in water conservancy projects through multi-source data fusion. This method enables deep fusion and precise analysis of heterogeneous multi-source data. While automatically identifying defect types, it also completes three-dimensional geometric reconstruction and quantitative assessment of defects. Furthermore, the reconstruction results conform to the laws of material physics, improving the automation, accuracy, and engineering reliability of the detection process.

[0007] In a first aspect, embodiments of this application provide a method for identifying and quantitatively assessing defects in water conservancy projects through multi-source data fusion, the method comprising:

[0008] Acquire multimodal inspection data, including visible light images, infrared thermal imaging data, laser 3D point cloud data, and acoustic echo signals;

[0009] A topological graph structure is constructed from the laser 3D point cloud data, and a spatiotemporal graph convolutional network is used to extract spatial topological features that characterize the geometric shape and connection relationship of the engineering structure.

[0010] Based on the spatial topological features, the time-frequency features extracted from the acoustic echo signal by wavelet packet energy entropy are fused with the visual features extracted from the visible light image and infrared thermal imaging data to obtain multimodal fusion features.

[0011] A sequence extraction module based on the Transformer architecture is used to model the multimodal fusion features and extract defect-sensitive spatiotemporal segments containing spatial location and time interval information.

[0012] Based on the defect-sensitive spatiotemporal segment, a decoder that integrates physical information neural networks is used to reconstruct the three-dimensional structure of the defect and obtain a three-dimensional structural model of the defect. The decoder is configured to simultaneously satisfy the constraints of the observation data and the constraints of the physical laws characterized by the partial differential equations of mechanics of materials during the reconstruction process.

[0013] Based on the defect-sensitive spatiotemporal segment and the defect three-dimensional structural model, defect type determination and engineering status quantitative analysis are performed, and defect type identification results, defect three-dimensional structural model and quantitative evaluation report are output.

[0014] Secondly, embodiments of this application provide a multi-source data fusion system for identifying and quantitatively evaluating defects in water conservancy projects, applied to the multi-source data fusion method for identifying and quantitatively evaluating defects in water conservancy projects as described in the first aspect. The system includes:

[0015] The data acquisition module is used to acquire multimodal inspection data, including visible light images, infrared thermal imaging data, laser 3D point cloud data, and acoustic echo signals.

[0016] The topology feature extraction module is used to construct a topology graph structure from the laser 3D point cloud data and to extract spatial topology features that characterize the geometric shape and connection relationship of the engineering structure using a spatiotemporal graph convolutional network.

[0017] The multimodal fusion module is used to perform cross-modal fusion of the time-frequency features extracted from the acoustic echo signal by wavelet packet energy entropy with the visual features extracted from the visible light image and infrared thermal imaging data, based on the spatial topological structure features, to obtain multimodal fused features.

[0018] The spatiotemporal segment extraction module is used to model the multimodal fusion features using a sequence extraction module based on the Transformer architecture, and extract defect-sensitive spatiotemporal segments containing spatial location and time interval information;

[0019] The three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the defect based on the defect-sensitive spatiotemporal segment using a decoder that integrates physical information neural networks to obtain a three-dimensional structural model of the defect. The decoder is configured to simultaneously satisfy the constraints of observation data and the constraints of physical laws characterized by the partial differential equations of mechanics of materials during the reconstruction process.

[0020] The analysis output module is used to determine the defect type and perform quantitative analysis of the engineering status based on the defect-sensitive spatiotemporal segment and the defect three-dimensional structural model, and output the defect type identification result, the defect three-dimensional structural model and the quantitative evaluation report.

[0021] Thirdly, embodiments of this application provide an electronic device, including:

[0022] processor;

[0023] Memory used to store processor-executable instructions;

[0024] The processor is configured to implement the method for identifying and quantitatively evaluating defects in water conservancy projects by multi-source data fusion as described in the first aspect when executing the instructions.

[0025] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program that instructs a device to execute the multi-source data fusion method for identifying and quantitatively evaluating defects in water conservancy projects as described in the first aspect.

[0026] Compared with existing technologies, the advantages of this invention are as follows: By using a spatiotemporal graph convolutional network and a multi-head mutual attention mechanism, it achieves deep and adaptive fusion of multi-source data such as laser point clouds, visible light, infrared, and acoustic data at the spatial and semantic levels, overcoming the limitations of single-modality or simple splicing. It creatively introduces physical information neural networks into defect 3D reconstruction, generating 3D models that conform to both the detection data and the laws of material mechanics, achieving accurate and reliable quantification of defect size, location, and morphology. It utilizes Transformer and memory enhancement mechanisms to achieve intelligent defect perception and recognition, and combines meta-learning to achieve environmental adaptation of the model, improving the system's intelligence level and engineering applicability. It provides a complete solution from synchronous acquisition and registration of multi-source data to the generation of the final 3D evaluation report, and considers lightweight design, showing good prospects for field application. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the method for identifying and quantitatively evaluating defects in water conservancy projects using multi-source data fusion, as provided in an embodiment of this application.

[0028] Figure 2 This is a functional module architecture diagram of a multi-source data fusion system for water conservancy project defect identification and quantitative assessment provided in an embodiment of this application.

[0029] Figure 3 This is a schematic diagram of the hardware structure of an electronic device for performing the method or system according to an embodiment of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0031] It should be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0032] Based on the embodiments described in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] Example 1

[0034] Figure 1This is a flowchart illustrating the overall process of a multi-source data fusion method for identifying and quantitatively assessing defects in water conservancy projects, provided in one embodiment of this application. Figure 1 As shown, a method for identifying and quantitatively assessing defects in water conservancy projects using multi-source data fusion includes:

[0035] S110. Acquire multimodal inspection data, including visible light images, infrared thermal imaging data, laser 3D point cloud data, and acoustic echo signals. This step is fundamental to the entire method and is responsible for simultaneously collecting multiple types (modalities) of raw sensing data from the hydraulic engineering structure under inspection. The aim is to establish a comprehensive dataset encompassing visual appearance (visible light), temperature distribution (infrared), precise 3D geometry (laser point cloud), and internal acoustic response (acoustic echo), providing complete input for subsequent fusion analysis.

[0036] Specifically, in this embodiment, the steps for acquiring multimodal inspection data include: using a UAV equipped with a multi-sensor synchronous acquisition unit to simultaneously acquire visible light images, infrared thermal imaging data, and laser 3D point cloud data of the same area of ​​the target hydraulic engineering structure; after completing the UAV aerial survey acquisition, for the target area, the inspection personnel use a handheld acoustic detection device to perform gridded tapping on the corresponding positions of the structure surface according to the UAV flight path and the generated 3D model to collect acoustic echo signals. The visible light images, infrared thermal imaging data, and laser 3D point cloud data are spatiotemporally registered through timestamp alignment and spatial coordinate transformation to form a first registration dataset; based on the spatial coordinate information in the laser 3D point cloud data, the acquisition points of the acoustic echo signals are mapped to the first registration dataset, thereby establishing a spatial correspondence between the acoustic acquisition points and the 3D structure surface, completing the full-modal data registration.

[0037] Taking the lining section of a water conveyance tunnel in a water conservancy project as an example, firstly, a drone equipped with a high-precision POS system and a multi-sensor integrated pod (the pod integrates a visible light camera, an infrared thermal imager, and a lidar) is used to fly at a constant speed along the tunnel axis, simultaneously collecting high-definition visible light images, infrared thermal imaging data, and high-density laser 3D point cloud data of the same cross-section of the lining. The flight trajectory is controlled by pre-planned waypoints to ensure full coverage. Subsequently, inside the tunnel, inspection personnel use a dedicated tapping detection device equipped with a high-sensitivity microphone to arrange 10cm×10cm gridded measuring points on the lining surface according to the drone's flight path. A standardized force hammer is used to tap each point with a constant force, and the acoustic echo signal of each measuring point is recorded simultaneously.

[0038] During data processing, photogrammetry and 3D modeling software were first used to automatically register and reconstruct 3D images from visible light, infrared, and laser point clouds based on the timestamps and spatial coordinates recorded by the POS system. This generated a 3D model with texture mapping and temperature information as the first registration dataset. Finally, in the point cloud processing software, each acoustic measurement point collected from the ground was precisely projected onto the corresponding surface position of the 3D model based on the centimeter-accurate 3D coordinates provided by the laser point cloud. This established a strict correspondence between acoustic signals and specific 3D spatial coordinates and visual features, forming a full-modal registration database covering light, heat, geometry, and acoustics, laying a solid spatially consistent foundation for subsequent fusion analysis.

[0039] S120. Construct a topological graph structure from the laser 3D point cloud data, and use a spatiotemporal graph convolutional network (ST-GCN) to extract spatial topological features representing the geometric shape and connectivity of the engineering structure. This step uses the highest-precision laser 3D point cloud data as the core, constructing a topological graph and utilizing a spatiotemporal graph convolutional network (ST-GCN) to transform discrete point cloud data into structured, learnable features. Its core function is to accurately represent the geometric shape of the engineering structure itself, the spatial connectivity between components, and dynamic change trends, providing a unified and accurate spatial reference and structural context for all subsequent multimodal features.

[0040] Specifically, in this embodiment, the steps of constructing a topological graph structure from laser 3D point cloud data and extracting spatial topological features include:

[0041] The laser 3D point cloud data is preprocessed, including noise reduction, outlier removal, and voxel downsampling.

[0042] Construct a k-dimensional tree spatial index, and based on the spatial index, use the k-nearest neighbor algorithm to establish local neighborhood connections for each point cloud node;

[0043] Define a node feature vector that includes at least the node's three-dimensional coordinates, normal vector, curvature, and local point density;

[0044] Based on the local neighborhood connections and node feature vectors, an undirected weighted topological graph representing the geometric connection relationship of the engineering structure is constructed, where nodes represent point cloud data points, edges represent the spatial adjacency relationship between nodes, and the edge weights are jointly determined by the Euclidean distance and feature similarity between nodes.

[0045] The topology graph is input into a spatiotemporal graph convolutional network. This network aggregates the features of neighboring nodes in the spatial dimension through graph convolution operations and captures the dynamic evolution pattern of structural features in the temporal dimension through temporal convolution operations, thereby extracting the spatial topological features of the engineering structure.

[0046] Taking the concrete overflow surface of a gravity dam as an example, the high-density point cloud acquired by UAV laser scanning is first preprocessed. Outlier filtering removes drifting points, and voxel mesh downsampling reduces the data volume while preserving the main geometric features. Subsequently, a spatial index tree is constructed to quickly find the neighboring points of each point cloud data point, establishing local connectivity. A comprehensive feature vector is calculated for each point, including its spatial location, surface orientation, local curvature, and the density of surrounding points. Based on these local connections and node features, a weighted topology graph is constructed. The connections between points in the graph consider not only spatial distance but also the similarity of their surface features; points that are close and have similar features have stronger connections. Finally, this topology graph, representing the complex surface geometry and connectivity of the overflow surface, along with time-series data from multiple inspections, is fed into a spatiotemporal graph convolutional neural network. Through learning, the network can understand the morphological relationships of different regions of the overflow surface (such as the nose sill and the reverse arc segment) in space, capture the expansion trend of defects such as surface erosion and concrete spalling in time, and finally extract spatial topological features that can comprehensively reflect the structural health status and dynamic changes of the overflow surface.

[0047] S130. Based on the spatial topological features, the time-frequency features extracted from the acoustic echo signal using wavelet packet energy entropy are fused across modes with the visual features extracted from the visible light image and infrared thermal imaging data to obtain a multimodal fusion feature. This step is the core fusion step of the entire method. Using the precise spatial topology obtained in the previous step as a benchmark, the acoustic time-frequency features reflecting the internal state of the material (such as wavelet packet energy entropy) are deeply integrated with the visual features reflecting surface texture and temperature anomalies. By constructing a multi-head mutual attention fusion module, using visual features as queries and acoustic features as keys and values, cross-modal attention weights are calculated to achieve semantic-level alignment between acoustic property changes and visual anomalies. Based on this, an adaptive gating mechanism is introduced to dynamically adjust the contribution of each modal feature, ultimately generating a spatiotemporal frequency domain deep fusion feature containing richer and more robust defect information.

[0048] Specifically, in this embodiment, the step of cross-modal fusion of acoustic time-frequency features and visual features includes:

[0049] Visual Feature Extraction: A lightweight RepVGG backbone network with depthwise separable convolution and attention enhancement is employed to extract multi-scale visual features from the visible light image and infrared thermal imaging data, respectively. Spatial pyramid pooling is then used to obtain a visual feature tensor with geometric semantics. Specifically, visual feature extraction is implemented using the PyTorch framework deployed on a mobile workstation, employing a lightweight RepVGG-A0 backbone network with depthwise separable convolution and attention enhancement. The network first scales the input visible light image and infrared thermal imaging to 512×512 pixels and performs normalization. The RepVGG network, through its unique reparameterization structure, efficiently extracts multi-scale texture and edge features from the visible light image, and temperature gradient and hotspot features from the infrared image. Subsequently, spatial pyramid pooling layers aggregate these multi-scale features into four fixed-size feature maps (scales of 1×1, 2×2, 3×3, and 6×6), which are then concatenated and subjected to 1×1 convolution for dimensionality reduction, ultimately outputting a 256-dimensional visual feature tensor that incorporates spatial geometric semantics.

[0050] Acoustic time-frequency feature extraction: The acoustic echo signal is subjected to multi-level wavelet packet decomposition, and the wavelet packet energy entropy of multiple sub-bands at a specified decomposition level is calculated to form a time-frequency feature vector. Based on the spatial topological features, the spatial coordinates of the acoustic acquisition points are mapped to the three-dimensional structural surface to form an acoustic feature distribution with spatial location attributes. Specifically, the acoustic time-frequency feature extraction is completed in a MATLAB and Python hybrid programming environment. For the acoustic echo signal (sampling rate 200kHz) acquired at each impact point, a 7-level 'db10' wavelet basis wavelet packet decomposition is first performed. The wavelet packet energy entropy of all 128 sub-bands (each with a bandwidth of approximately 781.25Hz) at the 7th level is calculated, and 16 specific frequency bands (mainly concentrated in the 0-12.5kHz range) with concentrated energy and sensitive to concrete voids are selected to form the time-frequency feature vector. Based on the previously extracted spatial topological features (i.e., the 3D point cloud model), the nearest neighbor search algorithm is used to accurately map the two-dimensional measurement coordinates of each acoustic acquisition point to the corresponding three-dimensional structural surface coordinates, thereby giving the abstract acoustic feature vector a clear three-dimensional spatial position attribute and forming an acoustic feature distribution that corresponds one-to-one with the nodes of the 3D model.

[0051] Cross-modal attention fusion: A multi-head mutual attention fusion module is constructed, using the visual feature tensor as the query and the acoustic feature distribution as the key and value. A cross-modal attention weight matrix is ​​calculated, and a deep semantic association between acoustic material stiffness variation features and image structure texture and temperature anomaly features is established through a soft alignment mechanism. Specifically, cross-modal attention fusion is implemented by a custom multi-head mutual attention module built on the TensorFlow framework. This module reshapes the visual feature tensor obtained in the previous step into a query matrix and the acoustic feature distribution with spatial location into a key matrix and a value matrix. Through eight parallel attention heads, the cross-modal correlation between the visual query and the acoustic key is calculated respectively, generating the attention weight matrix. This process achieves soft alignment; for example, high-entropy features in the acoustic signal indicating local material porosity (stiffness reduction) are automatically associated with microcrack textures at corresponding locations in the visible light image and temperature anomaly regions shown in the infrared image, thereby establishing a deep correspondence between the internal material state and the external appearance and thermal anomalies at the semantic level.

[0052] Specifically, the cross-modal gating attention of spatial topology awareness is represented as follows:

[0053] ,

[0054] ,

[0055] ,

[0056] in, Represents the visual feature matrix (from visible light and infrared images). This represents the acoustic time-frequency feature matrix (derived from wavelet packet energy entropy extraction). Represents the topological graph structure. Indicates the first Layer graph convolution operations extract topological features. Indicates the number of graph convolution layers (e.g.) ), Represents the topology enhancement matrix. Represents the dimension of the key vector (i.e.) The number of columns is used to scale the attention score and prevent gradient vanishing. The original attention score matrix representing visual and acoustic features. This represents the row normalization function, which transforms the attention scores into a probability distribution. A global pooling vector representing visual features (such as average pooling). Represents the gated weight matrix (which can be learned). This represents the gating bias vector. The matrix represents the cross-modal attention weights for topology enhancement, indicating the semantic association strength between each visual and acoustic location, incorporating structural topology information. The circle (⊙) represents the element-wise multiplication (Hadamard product), used to broadcast the gated vectors to the matrix. Represents the node feature matrix, Represents the graph convolution projection matrix. This represents the visual gating vector.

[0057] Adaptive Feature Weighting: Based on the cross-modal attention weight matrix, the aligned acoustic and visual features are adaptively weighted and fused. A gating mechanism is introduced to dynamically adjust the contribution of each modal feature in the fused output, generating a multimodal fusion feature with deep spatiotemporal frequency domain fusion. Specifically, adaptive feature weighting is performed after cross-modal attention fusion. Based on the calculated cross-modal attention weight matrix, the system performs preliminary fusion of the acoustic and visual features that have been aligned through the attention mechanism. Subsequently, a learnable gating network is introduced. This network receives the original acoustic and visual features as input and outputs a dynamic weight vector. This gating mechanism can determine the reliability of each modal data according to the specific scenario. For example, when the lighting conditions are extremely poor, causing the visible light image to be blurred, the contribution of visual features is automatically reduced, while the weights of acoustic and infrared features are increased; conversely, when the surface is clean and the acoustic signal is interfered with by environmental noise, the proportion of visual features is increased. Finally, the weighted fusion after gating adjustment generates a robust and complementary deep spatiotemporal frequency domain fusion feature, which serves as the input for subsequent precise defect localization.

[0058] S140. A sequence extraction module based on the Transformer architecture is used to model the multimodal fusion features, extracting defect-sensitive spatiotemporal segments containing spatial location and temporal interval information. This step is equivalent to an intelligent reconnaissance and focusing process. Using a Transformer-based sequence model, segments exhibiting significant anomalies in both time and space are automatically identified and located in the fused multimodal feature sequence. Its function is to efficiently and accurately filter out the most likely suspected defect-containing areas and their spatiotemporal ranges from massive amounts of data, providing precise target input for subsequent high-cost 3D reconstruction and greatly improving overall processing efficiency.

[0059] Specifically, in this embodiment, the step of extracting defect-sensitive spatiotemporal segments includes:

[0060] S1401. The multimodal fusion features are rearranged along the time dimension and the spatial topology node dimension to construct a spatiotemporal feature sequence, where each sequence element is associated with a specific spatial node and a timestamp. Specifically, the multimodal fusion features are rearranged along the time dimension and the spatial topology node dimension to construct a spatiotemporal feature sequence. In specific implementation, taking five consecutive periodic inspections of the water conveyance tunnel lining section as an example, each inspection acquires fusion features of approximately 5000 spatial nodes (corresponding to point clouds). First, the data of these five time steps are arranged in chronological order. Then, within each time step, the feature vectors of all nodes are concatenated according to the pre-established topology graph node order. Finally, a three-dimensional tensor of shape (5, 5000, feature dimension) is formed, and it is reshaped into a feature sequence of length 25000. Each element in the sequence uniquely corresponds to a specific spatial node and a specific timestamp (e.g., the 3rd inspection, node number 1205), thereby uniformly encoding the spatiotemporal information.

[0061] S1402. Learnable positional encoding is added to the spatiotemporal feature sequence, and the long sequence is divided into temporal segments with engineering semantics based on the functional partitioning of the engineering structure and the detection time interval. Specifically, learnable positional encoding is added to the spatiotemporal feature sequence, and the long sequence is divided into temporal segments with engineering semantics based on the functional partitioning of the engineering structure and the detection time interval. Specifically, firstly, the nn.Embedding layer in PyTorch is used to generate a learnable encoding vector for each position in the sequence. Then, according to the structural characteristics of the water conveyance tunnel (such as the inlet section, tunnel body section, and outlet section), the spatial nodes are grouped, and combined with the time information of one week between two inspections, the long sequence is divided into multiple independent temporal segments. For example, the data of all nodes belonging to the tunnel body lining area within three consecutive inspections (three weeks) are combined into one segment, so that each segment represents the state evolution of a specific structural part within a continuous time period, giving it a clear engineering physical meaning.

[0062] S1403. Construct a multi-layer Transformer encoder module to model the global dependencies of features within the time-series segment using a self-attention mechanism, thereby capturing long-range spatiotemporal correlations. Specifically, a multi-layer Transformer encoder module is constructed to model the global dependencies of features within the time-series segment using a self-attention mechanism, thereby capturing long-range spatiotemporal correlations. In this embodiment, a standard 6-layer Transformer encoder is built, with a hidden layer dimension of 512 and 8 attention heads. After inputting a pre-divided segment (e.g., containing 1500 spatiotemporal elements), the self-attention mechanism in the encoder allows any element within the segment (e.g., week 2, a crack tip node) to directly calculate correlation weights with all other elements within the segment (including other position nodes from the previous or following week), thus enabling the modeling of complex long-range spatiotemporal dependencies such as crack propagation leading to changes in the distal stress field.

[0063] S1404. A lightweight evaluation head is connected after the output layer of the Transformer encoder to calculate the defect sensitivity score of each temporal segment and select segments with scores higher than the preset sensitivity threshold as defect-sensitive spatiotemporal segments. Specifically, a lightweight evaluation head is connected after the output layer of the Transformer encoder to calculate the defect sensitivity score of each temporal segment and select segments with scores higher than the preset sensitivity threshold as defect-sensitive spatiotemporal segments. This evaluation head is implemented by a two-layer fully connected network, which receives the aggregated features of the entire segment (such as CLS tokens or features after average pooling) and outputs a scalar score. The score integrates the abnormal intensity, spatiotemporal consistency, and evolution trend of features within the segment. During the training phase, the threshold is learned through labeled data (such as segments with known defective regions); during the application phase, all segment scores are normalized by the Sigmoid function and compared with the preset threshold of 0.7. Segments with scores higher than this threshold (such as a score of 0.85) are judged to be defect-sensitive and proceed to the next step of processing.

[0064] S1405. For the selected defect-sensitive spatiotemporal segments, decode their corresponding spatial node sets and continuous time intervals, outputting a set of defect-sensitive spatiotemporal segments containing specific spatial locations and occurrence time windows. Specifically, for the selected defect-sensitive spatiotemporal segments, decode their corresponding spatial node sets and continuous time intervals, outputting a set of defect-sensitive spatiotemporal segments containing specific spatial locations and occurrence time windows. Each selected segment recorded its corresponding spatial node ID range and time step index during the original segmentation. The decoding process is a reverse mapping: based on the segment ID, query its contained list of spatial nodes (e.g., node IDs 2010 to 2150) and time step index (e.g., the 2nd to 4th inspections). The final output is a structured list, each entry containing: 1) a set of 3D point cloud coordinates, accurately outlining the spatial range of the suspected defect; 2) a time interval, indicating the time window in which the anomaly was detected. This result provides accurate spatiotemporal localization input for subsequent 3D reconstruction.

[0065] Furthermore, the Transformer encoder module is configured to perform multi-scale spatiotemporal modeling, specifically including:

[0066] In each layer of the Transformer encoder, two attention calculations are performed in parallel: one is to calculate the global self-attention of all spatial nodes within the current time segment across all time steps; the other is to calculate the local spatiotemporal window attention of each spatial node within its adjacent spatial neighborhood and neighboring time steps, using a preset local spatiotemporal window as the range. In a specific implementation, taking a time segment of the lining of a water conveyance tunnel section processed in layer L as an example, this segment contains the features of M spatial nodes over T consecutive time steps. The global attention head calculates the full connectivity between all M×T spatiotemporal units in this segment, aiming to capture long-range correlations such as the loosening of a bolt at a certain location and the generation of microcracks in distant concrete. The local window attention head uses a 3×3 spatiotemporal window (i.e., each node focuses on itself, one time step before and after it, and the eight nearest neighbors in the graph structure), aiming to finely capture short-range evolution details such as the slow extension of local cracks over two consecutive weeks. These two attention calculations are performed in parallel at the same layer, forming a complementary relationship.

[0067] The outputs of the global self-attention and the local spatiotemporal window attention are adaptively weighted and fused using a gating fusion mechanism to form the final output features of each layer. The gating mechanism is implemented by a lightweight gating network (such as a small feedforward neural network). This network receives the input features of the current layer and outputs a scalar gating value ranging from [0,1]. When calculating the final output of the Lth layer, this gating value is used to weight the global attention output, and (1 - the gating value) is used to weight the local attention output, and then the results are summed. This allows the network to dynamically determine the emphasis of information. For example, when processing regions with clear data and complete structure, the gating value approaches 1, relying more on the global context for consistency judgment; while when processing noisy data or suspected defect boundaries requiring fine-grained localization, the gating value approaches 0, focusing more on the fine-grained analysis of the local window.

[0068] The intermediate layer outputs of the multi-layer Transformer encoder undergo multi-scale feature fusion. Shallow layer outputs represent fine-grained local spatiotemporal anomaly patterns, while deep layer outputs represent coarse-grained global structural state evolution. These are fused with a feature pyramid structure through cross-layer connections. Specifically, a feature pyramid network-like architecture is adopted. The feature map of layer 2 (shallow layer) (high resolution, sensitive to minute, transient anomalies such as echo mutations) is dimensionality-reduced. Simultaneously, the feature map of layer 5 (deep layer) (low resolution, but rich in semantic information, sensitive to macroscopic, slowly developing trends such as concrete carbonation accumulation) is upsampled. Then, these features at different scales are fused by element-wise addition or concatenation. The fused features include both fine-grained spatiotemporal anomaly signals from the shallow layer (such as instantaneous high-frequency vibration anomalies at a certain point) and coarse-grained structural state trends from the deep layer (such as a slow decrease in stiffness across the entire abutment region over three months), thus achieving a comprehensive characterization of defects from microscopic to macroscopic levels.

[0069] In the final layer of the Transformer encoder, a learnable defect pattern memory query library is introduced. This library is a trainable embedding matrix, where each row stores a feature vector of a typical defect pattern extracted from historical data or expert knowledge, such as a uniform corrosion propagation pattern in rebar, a seepage crack pattern developing along construction joints, or a localized impact spalling pattern. When processing a new time segment, attention weights are calculated between the final feature representation of that segment and all pattern vectors in the memory library. The final output feature is a linear combination of the original feature and the memory feature weighted and summed by the attention weights. This process is equivalent to allowing the model to recall and compare known typical defect patterns before making a final judgment, thereby enhancing the specificity of identifying known defect patterns and improving the accuracy and interpretability of the judgment when faced with complex and mixed signals.

[0070] S150. Based on the defect-sensitive spatiotemporal segment, a decoder integrating a Physical Information Neural Network (PINN) is used to reconstruct the three-dimensional structure of the defect, obtaining a three-dimensional structural model of the defect. The decoder is configured during the reconstruction process to simultaneously satisfy both observational data constraints and physical constraints represented by the partial differential equations of mechanics of materials. This step is the key innovation and core of the method. Receiving the defect-sensitive segment located in the previous step, a decoder integrating a Physical Information Neural Network (PINN) is used to not only recover the three-dimensional geometry, location, and size of the defect from the data, but also to embed the laws of mechanics of materials as hard constraints into the reconstruction process. Its function is to generate a highly reliable three-dimensional digital model of the defect that conforms to both observational data and strictly follows physical principles, thereby achieving a leap from qualitative identification to quantitative reconstruction.

[0071] Specifically, in this embodiment, the step of using a decoder that integrates physical information neural networks to reconstruct the three-dimensional defects includes:

[0072] A multi-resolution geometric encoder-decoder architecture is constructed: a multi-resolution point cloud representation is built based on the set of spatial nodes in the defect-sensitive spatiotemporal segment. The encoder uses a hierarchical graph convolutional network to extract multi-scale geometric features, and the decoder uses a coordinate-based implicit neural network to map the geometric features and query point coordinates into a signed distance function value. When constructing the multi-resolution geometric encoder-decoder architecture, firstly, based on the suspected voided area delineated by the defect-sensitive spatiotemporal segment, a set of spatial nodes is extracted from the corresponding original point cloud, and a multi-resolution point cloud representation is constructed through downsampling at different voxel sizes (e.g., 5mm, 10mm, 20mm). The encoder uses a three-layer hierarchical graph convolutional network (based on the DeepGCN architecture) to aggregate neighborhood information at three resolution levels, extracting multi-scale geometric features from local details to the global region. The decoder is a coordinate-based implicit neural network (e.g., the SIREN network). This network takes the global feature vector output by the encoder and the coordinates of any three-dimensional spatial query point as input, and through multi-layer forward propagation, directly outputs the signed distance function value from the query point to the reconstructed defect surface (e.g., the interface of a concrete cavity). Positive values ​​indicate that the point is outside the solid, negative values ​​indicate that it is inside the cavity, and zero values ​​indicate the surface.

[0073] Embedded Physical Information Constraints: During the forward propagation of the decoder, the embedded linear elasticity control equations are solved simultaneously. These equations use the reconstructed geometry as the computational domain and material property parameters and boundary conditions as inputs to calculate the predicted values ​​of the physical fields. Taking the simulation of void defects inside concrete as an example, the defect surface predicted by the decoder (an isosurface with an SDF value of zero) is dynamically extracted into a triangular mesh using a moving cube algorithm, serving as the boundary of the computational domain for subsequent physical solutions. In the same forward propagation, the preset concrete elastic modulus, Poisson's ratio, and the load boundary conditions applied to the structure are input along with this geometric domain into a physical information network module specifically for solving the Navier-Lame equations. This module employs a physical information neural network architecture to automatically calculate the displacement and stress field distributions generated inside the structure under the given defect geometry assumptions and outputs the predicted values ​​of these physical fields at the sampling points. This process automatically and in real-time correlates abstract physical laws with specific geometric shapes.

[0074] The governing equations for the elastic dynamics of spatially variable material parameters are expressed as follows:

[0075] ,

[0076] ,

[0077] ,

[0078] in, Represents a spatial coordinate vector. Representing a time variable, it is used to describe the response process of defects under dynamic loads (such as water flow impact and vibration). This represents the displacement field vector function, which represents any point inside the material. In time The displacement that occurred This represents the density function of a material, and can indicate the density differences within concrete caused by pores and uneven aggregate distribution. This represents a fourth-order elastic tensor field, describing the material in spatial coordinates. The stress-strain relationship at the point includes information such as elastic modulus and Poisson's ratio. Represents the strain tensor. This represents the body force vector field, derived from gravity, inertial force, seepage pressure, etc. This represents a Material Parameter Prediction Network (MLP), with the global geometric feature vector as input. With coordinates Output elastic tensor , This represents a subset of the defect-sensitive point cloud. The formula supports dynamic response analysis, where material parameters are derived by a neural network based on global geometric features. Combined prediction with local coordinates, This represents the geometry encoder function.

[0079] A composite loss function comprising data fitting loss, physics consistency loss, and geometric regularization loss is defined for collaborative optimization. The data fitting loss calculates the difference between the signed distance function of the reconstructed surface and the anomalous observations extracted from the defect-sensitive spatiotemporal segment. The physics consistency loss calculates the residual between the physics field predicted by the decoder and the standard physics field obtained by numerically solving the governing equations. The geometric regularization loss constrains the curvature and smoothness of the reconstructed surface to ensure geometric rationality. The data fitting loss calculates the mean square error between the decoder's predicted value of the signed distance function and the true value (estimated or set from acoustic or visual anomalous observations in the defect-sensitive area). The physics consistency loss is crucial; it calculates the L2 norm of the residual between the stress field predicted by the physical information network and the standard stress field obtained by numerically simulating the same defect geometry using a traditional finite element solver (such as FEniCS), forcing the neural network's physical predictions to approximate physical laws. The geometric regularization loss penalizes drastic changes in surface curvature by calculating the Laplacian energy of the reconstructed surface mesh, ensuring that the reconstructed cavity or crack surface is smooth and physically plausible, rather than ill-conditioned and distorted. The total loss is a weighted sum of these three losses.

[0080] The formula for the physical consistency loss of energy functional constraints is:

[0081] ,

[0082] ,

[0083] in, The three-dimensional computational domain representing the reconstructed defect is a continuum space containing the defect and its affected region. Represents the computational domain Boundary surface, Represents a volume element. Represents an area element. This represents the stress and displacement fields predicted by the network. Represents the external load field. Represents the boundary operator, Indicates the boundary condition value. Indicates the boundary loss weights. Represents a physical information neural network, with spatial coordinates as input. Output displacement field . Represents the strain tensor. This represents the elastic tensor field predicted by the neural network, which is non-uniform and spatially varied, and is output by the material parameter network. This indicates a double dot product operation (tensor contraction). Representing stress divergence, the differential form of internal stress equilibrium. Represents the external force field. Represents the square of the L2 norm. This represents the volume integral (integration over the entire computational domain). This represents a surface integral (integrating over the entire boundary). The above formula uses a weak integral form, which is more suitable for irregular geometry and complex boundary conditions.

[0084] Alternating optimization training is performed using a two-stage training strategy. The first stage primarily uses forward data fitting to rapidly generate a preliminary geometric shape of the defect. The second stage fixes the geometric encoder parameters and iteratively optimizes the decoder parameters by backpropagating the physical field consistency loss through the physical information neural network layer. This ensures that the reconstructed 3D defect structure satisfies the observation data while its induced physical field response conforms to the laws of material mechanics. The alternating optimization training employs a two-stage strategy. In the first stage, the physical information network module is frozen, and the encoder and decoder are trained using only the data fitting loss and geometric regularization loss for approximately 5000 iterations. The goal of this stage is to rapidly generate a preliminary defect geometric shape that roughly matches the observed anomaly (such as a low-echo region) under data-driven conditions. In the second stage, the encoder parameters are fixed (geometric features are locked), the physical information network module is unfrozen, and the weight of the physical field consistency loss is significantly increased. In approximately 2000 iterations of this stage, the parameters of the decoder network and the physical information network are primarily optimized through backpropagation of the physical loss. This process is like physically refining a preliminary geometric model, constantly adjusting its shape until the stress distribution calculated by the physical information network under that shape is highly consistent with the results of ideal physical solutions, thereby ensuring that the final reconstructed three-dimensional defect model not only looks like a defect, but its existence also conforms to the inherent laws of materials mechanics.

[0085] Furthermore, specific implementations of embedded physical information constraints include:

[0086] Adaptive Coupled Modeling of the Physics Field: The implicit geometric representation generated by the decoder is dynamically transformed into a computational mesh for solving the physics field. This process employs an adaptive mesh refinement strategy, automatically refining the mesh in geometrically significant regions and using a sparse mesh in flat regions to achieve a balance between computational accuracy and efficiency. Adaptive coupled modeling of the physics field is implemented through an integrated mesh processing pipeline. After the decoder's implicit neural network outputs a signed distance field, the system first extracts an initial isosurface triangular mesh using the Marching Cubes algorithm. Subsequently, this initial mesh is fed into an adaptive mesh refinement library based on error estimation. The system solves a simplified test physics problem on the mesh, automatically identifying geometrically significant regions, such as predicted crack tips, hole edges, or stress concentration areas, based on the spatial gradient of the solution. In these regions, the mesh is recursively subdivided, reducing the triangle area to a preset minimum to capture details; while in geometrically flat regions with minimal changes in the physics field, a relatively sparse mesh is maintained. Ultimately, this adaptively optimized, high-quality mesh is seamlessly passed to the subsequent physics solver, ensuring computational accuracy in critical defect regions while avoiding excessive computational overhead in non-critical regions.

[0087] Spatial Heterogeneity Characterization of Material Parameters: In solving the governing equations of linear elasticity, the material property parameters are modeled as continuous functions of spatial coordinates rather than uniform constants. A parallel branch of the decoder is trained to output a spatially varying field of material parameters, which serves as input to the physical constraint solution to characterize the material property degradation caused by concrete aging or steel corrosion. The spatial heterogeneity characterization of material parameters is achieved by expanding the output dimension of the decoder. A parallel, structurally similar branch network is added outside the backbone network of the decoder. This branch network takes the same global geometric features and spatial query point coordinates as input, but its output is no longer the distance to the surface, but rather a vector of material properties at that point, such as the elastic modulus and Poisson's ratio. During training, this branch network is supervised by a physical field consistency loss and, if possible, multispectral or electrical inversion data, to learn to predict a spatially continuously varying field of material parameters. For example, when faced with a region of steel corrosion, this branch can predict the reduction in effective elastic modulus of the concrete around the expansion zone of corrosion products due to tensile cracking. The predicted non-uniform material field, along with the reconstructed geometry, is input into the physical information neural network as a boundary condition, enabling the physical simulation to more realistically reflect the changes in structural mechanical response caused by material degradation.

[0088] Uncertainty Quantification and Reliable Reconstruction: In the calculation of physical field consistency loss, an uncertainty estimation module based on a Bayesian neural network is introduced. This module simultaneously predicts the mean and variance of the physical field residuals and adds an uncertainty-aware loss term to the composite loss function. This guides the model to rely more on data fitting in regions of high physical constraint uncertainty and to strictly follow physical laws in regions of low uncertainty, thereby achieving a reliable reconstruction result with known risk. Uncertainty quantification and reliable reconstruction are achieved by replacing key layers of the physical information neural network with Bayesian layers. Specifically, in the physical information network module for calculating physical field consistency loss, the last few fully connected layers are replaced with Bayesian neural network layers using variational inference. During each forward propagation, these layers not only output the predicted physical field value but also simultaneously output the uncertainty variance of the prediction. When calculating the physical field consistency loss, not only is the residual between the predicted value and the reference value calculated, but also an uncertainty-aware loss term composed of this residual and the prediction uncertainty is calculated. During the second phase of training, when performing physical refinement, this loss term dynamically adjusts the strength of the physical constraints: in regions where the model's predictions are highly uncertain (e.g., where defect boundaries are blurred or where material degradation is severe and difficult to model accurately), this loss term is weakened, making the reconstruction result fit the original observation data more closely; while in regions where the model's predictions are highly certain, this loss term strengthens the constraints of physical laws. Ultimately, the reconstructed 3D defect model is accompanied by a confidence heatmap, clearly indicating which parts of the model have highly reliable geometry and material properties, and which parts have higher inference risks, thus providing more scientific risk warnings for engineering decisions.

[0089] Specifically, the formula for the uncertainty-aware adaptive reconstruction loss can be the variational Bayesian reconstruction loss, expressed as:

[0090] ,

[0091] ,

[0092] in, This represents latent random variables, characterizing the sources of uncertainty in the reconstruction process, such as: uncertainties in material parameters, ambiguities in geometry, sensor measurement noise, and uncertainties in boundary conditions. Indicates the first Individual observations (such as SDF, acoustic features). This represents the variational posterior distribution (parameterized probability distribution), which is usually assumed to be a diagonal Gaussian distribution. Variational parameters (mean) and logarithmic variance ), output by a neural network, Indicates dependence on random variables The model's predicted values, This represents the variance of the forecast uncertainty. This represents the prior distribution (such as a Gaussian mixture model). Indicates the Kullback-Leibler divergence. This indicates a trade-off with hyperparameters. This indicates the number of observation points. Represents the uncertainty-weighted residual term, when When the uncertainty is large, the weight of this item is small, allowing for a larger error. When the uncertainty is small (low), the weight of this term is large, forcing an exact fit. This represents a regularization term for uncertainty, preventing the model from growing indefinitely. To evade the responsibility of fitting. The negative logarithmic form representing the likelihood of conditional data. Represents the physical consistency loss (depending on) That is, the physical consistency loss of the preceding energy functional constraints. But all field quantities ( All depend on : , Represents the variational distribution The mathematical expectation is given. This formula uses the uncertainty variance as an adaptive weight to achieve risk-sensitive reconstruction.

[0093] S160. Based on the defect-sensitive spatiotemporal segment and the defect 3D structural model, perform defect type determination and engineering status quantitative analysis, and output defect type identification results, defect 3D structural model, and quantitative assessment report. This step is the final decision-making and delivery stage of the method. Based on the located defect segment and the reconstructed 3D model, intelligently determine the type of defect (such as cracks, voids, corrosion), and calculate key quantitative parameters (such as length, width, depth, and volume) in conjunction with the 3D model. Its function is to integrate all intermediate analysis results and generate a structured, visualized comprehensive report that can be directly used for engineering diagnosis, risk assessment, and maintenance decisions, completing the closed loop from raw data to engineering knowledge.

[0094] The multimodal inspection data also includes geographic information data and meteorological data. The method further includes adaptively adjusting the model parameters of the sequence extraction module or the decoder based on the geographic information data and meteorological data through a meta-learning framework to improve recognition performance under different environments. Taking the inspection of two concrete aqueducts in different river basins in northern and southern my country as an example, this embodiment constructs an environment adaptive system based on meta-learning, addressing the characteristics of aqueduct surfaces being prone to moss growth and infrared features being greatly affected by humidity in the rainy and humid environment of the south, and the characteristics of aqueducts being prone to freeze-thaw cycles and salt erosion in the dry and cold environment of the north. The system first encodes the aqueduct's geographical location (latitude, longitude, and altitude), recent temperature and humidity, precipitation, wind speed, and other data obtained from GIS and meteorological stations into an environmental feature vector. During the model deployment phase, when the system identifies the current detection target as the northern aqueduct-winter pattern, the meta-learner quickly adjusts the self-attention weight distribution of the Transformer in the sequence extraction module based on the environmental vector, making it pay more attention to specific textures and micro-displacement patterns caused by frost heave. Simultaneously, it dynamically adjusts the initial bias of the material parameter prediction branch in the decoder, prioritizing low-temperature brittleness and salt corrosion degradation models when solving the physical information neural network. Through this lightweight, environment- and task-related parameter adjustment of the pre-trained model, after fine-tuning with a small number of field samples from northern winters, the system's accuracy in identifying freeze-thaw cracks and salt corrosion spalling is improved by more than 30% compared to directly applying a general model, significantly enhancing the method's robustness and generalization ability under different geographical and climatic conditions.

[0095] Example 2

[0096] like Figure 2 As shown, this application provides a multi-source data fusion system for water conservancy engineering defect identification and quantitative assessment 200. This system 200 is deployed on a server or edge computing device in a software, hardware, or a combination of both, and includes the following collaborative functional modules:

[0097] The data acquisition module 210 is responsible for scheduling sensors such as drones and acoustic equipment to complete the synchronous or sequential acquisition and uploading of multimodal data, and provides preliminary spatiotemporal registration functions.

[0098] The topology feature extraction module 220 receives laser point cloud data from the data acquisition module 210, performs point cloud preprocessing and topology graph construction, and extracts spatial topology features through a spatiotemporal graph convolutional network.

[0099] The multimodal fusion module 230, based on the spatial topological structure features output by the topological feature extraction module 220, receives visual data and acoustic data, and performs cross-modal attention fusion and adaptive weighting as described in S130 of Embodiment 1 of this specification and subsequent specific implementations to generate multimodal fusion features.

[0100] The spatiotemporal segment extraction module 240 reassembles the feature sequence output by the multimodal fusion module 230, uses a Transformer-based sequence model for modeling and evaluation, and filters and outputs defect-sensitive spatiotemporal segments.

[0101] The 3D reconstruction module 250, targeting the suspected defect area located by the spatiotemporal segment extraction module 240, calls the decoder of the fusion physical information neural network to perform high-precision 3D reconstruction with physical law constraints, generating a 3D structural model of the defect.

[0102] The analysis output module 260 integrates the information from the defect-sensitive spatiotemporal segments with the model generated by the 3D reconstruction module 250 to intelligently determine the defect type and calculate quantitative parameters, ultimately generating a comprehensive output that includes the identification results, a 3D visualization model, and a quantitative evaluation report.

[0103] The modules are sequentially called through well-defined data interfaces, forming an automated processing pipeline from raw data to evaluation reports.

[0104] Figure 3 This is an electronic device provided in one embodiment of this application. For example... Figure 3 As shown, the electronic device includes at least the following components: processor 301 and memory 300, communication interface 303, and bus 302.

[0105] In this embodiment of the application, memory 300 is used to store executable instructions of processor 301, which, when configured to execute instructions, implements the method as described in the first aspect.

[0106] In embodiments of this application, a computer-readable storage medium includes instructions that instruct a device to perform the method as described in the first aspect. For example, the instructions instruct the device to perform... Figure 1 The method is shown in the process steps.

[0107] In one embodiment of this application, the program operating in the electronic device may be a program that controls a central processing unit (CPU) or similar device to achieve the functions of the above-described embodiments of the present invention (a program that enables the computer to function). Information processed by these systems is then temporarily stored in random access memory (RAM) during processing, and subsequently stored in various ROMs such as read-only memory (FlashROM) and hard disk drives (HDDs), and read, corrected, and written by the CPU as needed.

[0108] It should be noted that a portion of the electronic device described above can also be implemented using a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and the program recorded on the recording medium can be read into the computer and executed.

[0109] It should be noted that the computer mentioned here refers to a computer built into an electronic device, employing hardware including an operating system and peripheral devices. Furthermore, computer-readable recording media refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage systems such as hard drives built into the computer.

[0110] Furthermore, computer-readable recording media can include: media that dynamically stores programs for short periods of time, such as communication lines used when transmitting programs via networks like the Internet or communication lines like telephone lines; and media that store programs for fixed periods of time, such as volatile memory inside a computer that serves as a server or client in this case. In addition, the aforementioned program can be a program used to implement the above-mentioned functions, or it can be a program that can implement the above-mentioned functions by combining them with programs already recorded in the computer.

[0111] Furthermore, the electronic device in the above embodiments can also be implemented as an assembly (system group) composed of multiple systems. Each system constituting the system group can possess some or all of the functions or functional blocks of the electronic device in the above embodiments. As a system group, it is sufficient to have all the functions or functional blocks of the electronic device.

[0112] Those skilled in the art should recognize that the above embodiments are only used to illustrate this application and are not intended to limit this application. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of this application fall within the scope of protection claimed in this application.

Claims

1. A method for identifying and quantitatively assessing defects in water conservancy projects using multi-source data fusion, characterized in that, Includes the following steps: Acquire multimodal inspection data, including visible light images, infrared thermal imaging data, laser 3D point cloud data, and acoustic echo signals; A topological graph structure is constructed from the laser 3D point cloud data, and a spatiotemporal graph convolutional network is used to extract spatial topological features that characterize the geometric shape and connection relationship of the engineering structure. Based on the spatial topological features, the time-frequency features extracted from the acoustic echo signal by wavelet packet energy entropy are fused with the visual features extracted from the visible light image and infrared thermal imaging data to obtain multimodal fusion features. A sequence extraction module based on the Transformer architecture is used to model the multimodal fusion features and extract defect-sensitive spatiotemporal segments containing spatial location and time interval information. Based on the defect-sensitive spatiotemporal segment, a decoder that integrates physical information neural networks is used to reconstruct the three-dimensional structure of the defect and obtain a three-dimensional structural model of the defect. The decoder is configured to simultaneously satisfy the constraints of the observation data and the constraints of the physical laws characterized by the partial differential equations of mechanics of materials during the reconstruction process. Based on the defect-sensitive spatiotemporal segment and the defect three-dimensional structural model, defect type determination and engineering status quantitative analysis are performed, and defect type identification results, defect three-dimensional structural model and quantitative evaluation report are output.

2. The method according to claim 1, characterized in that, The specific steps for obtaining multimodal inspection data include: A drone equipped with a multi-sensor synchronous acquisition unit was used to simultaneously acquire visible light images, infrared thermal imaging data, and laser 3D point cloud data of the same area of ​​the target hydraulic engineering structure. Using a handheld or fixed acoustic detection device, the structural surface corresponding to the location where the UAV collects data is tapped in a grid pattern to collect acoustic echo signals. The visible light image, infrared thermal imaging data and laser 3D point cloud data are spatiotemporally registered by timestamp alignment and spatial coordinate transformation to form the first registration dataset. Based on the spatial coordinate information in the laser 3D point cloud data, the acquisition points of the acoustic echo signal are mapped to the first registration dataset, thereby establishing a spatial correspondence between the acoustic acquisition points and the 3D structural surface and completing the full modal data registration.

3. The method according to claim 1, characterized in that, The specific steps for constructing a topological graph structure and extracting spatial topological features from laser 3D point cloud data include: The laser 3D point cloud data is preprocessed, including noise reduction, outlier removal, and voxel downsampling. Construct a k-dimensional tree spatial index, and based on the spatial index, use the k-nearest neighbor algorithm to establish local neighborhood connections for each point cloud node; Define a node feature vector that includes at least the node's three-dimensional coordinates, normal vector, curvature, and local point density; Based on the local neighborhood connections and node feature vectors, an undirected weighted topological graph representing the geometric connection relationship of the engineering structure is constructed, where nodes represent point cloud data points, edges represent the spatial adjacency relationship between nodes, and the edge weights are jointly determined by the Euclidean distance and feature similarity between nodes. The topology graph is input into a spatiotemporal graph convolutional network. This network aggregates the features of neighboring nodes in the spatial dimension through graph convolution operations and captures the dynamic evolution pattern of structural features in the temporal dimension through temporal convolution operations, thereby extracting the spatial topological features of the engineering structure.

4. The method according to claim 1, characterized in that, The specific steps for cross-modal fusion of acoustic time-frequency features and visual features include: Visual feature extraction: A lightweight RepVGG backbone network with depthwise separable convolution and attention enhancement is used to extract multi-scale visual features from the visible light image and infrared thermal imaging data, respectively, and obtains a visual feature tensor with geometric semantics through spatial pyramid pooling. Acoustic time-frequency feature extraction: The acoustic echo signal is decomposed into multiple wavelet packets, and the wavelet packet energy entropy of multiple sub-bands at a specified decomposition level is calculated to form a time-frequency feature vector; Based on the spatial topological features, the spatial coordinates of the acoustic acquisition points are mapped to the three-dimensional structural surface to form an acoustic feature distribution with spatial location attributes. Cross-modal attention fusion: Construct a multi-head mutual attention fusion module, using the visual feature tensor as the query and the acoustic feature distribution as the key and value, calculate the cross-modal attention weight matrix, and establish a deep semantic association between the acoustic material stiffness variation features and the image structure texture and temperature anomaly features through a soft alignment mechanism; Adaptive feature weighting: Based on the cross-modal attention weight matrix, the aligned acoustic and visual features are adaptively weighted and fused, and a gating mechanism is introduced to dynamically adjust the contribution of each modal feature in the fusion output, generating multimodal fusion features with deep spatiotemporal frequency domain fusion.

5. The method according to claim 1, characterized in that, The specific steps for extracting defect-sensitive spatiotemporal segments include: The multimodal fusion features are rearranged along the time dimension and the spatial topology node dimension to construct a spatiotemporal feature sequence, where each sequence element is associated with a specific spatial node and a timestamp; Learnable positional codes are added to the spatiotemporal feature sequences, and the long sequences are divided into temporal segments with engineering semantics based on the functional partitions of the engineering structure and the detection time interval. A multi-layer Transformer encoder module is constructed, and a self-attention mechanism is used to model the global dependency relationship of features within the time segment in order to capture long-distance spatiotemporal correlation. A lightweight evaluation head is connected after the output layer of the Transformer encoder to calculate the defect sensitivity score of each time segment, and to filter out segments with scores higher than the preset sensitivity threshold as defect sensitive spatiotemporal segments based on the preset sensitivity threshold. For the selected defect-sensitive spatiotemporal segments, decode their corresponding set of spatial nodes and continuous time intervals, and output a set of defect-sensitive spatiotemporal segments containing specific spatial locations and occurrence time windows.

6. The method according to claim 5, characterized in that, The Transformer encoder module is configured to perform multi-scale spatiotemporal modeling, specifically including: In each layer of the Transformer encoder, two types of attention calculations are performed in parallel: one is to calculate the global self-attention of all spatial nodes in the current time segment at all time steps; the other is to calculate the local spatiotemporal window attention of each spatial node in its adjacent spatial neighborhood and neighboring time steps within a preset local spatiotemporal window. The outputs of the global self-attention and the local spatiotemporal window attention are adaptively weighted and fused using a gating fusion mechanism to form the final output features of each layer. The intermediate layer outputs of the multi-layer Transformer encoder are subjected to multi-scale feature fusion, wherein the shallow layer outputs represent fine-grained local spatiotemporal anomaly patterns, the deep layer outputs represent coarse-grained global structural state evolution, and are fused with the feature pyramid structure through cross-layer connections. In the last layer of the Transformer encoder, a learnable defect pattern memory query library is introduced. This query library predefines several typical defect evolution pattern feature vectors, and uses an attention mechanism to match and weight the features of the current time segment with the query library to enhance the specificity of recognizing known defect patterns.

7. The method according to claim 1, characterized in that, The specific steps for 3D defect reconstruction using a decoder that integrates physical information neural networks include: Constructing a multi-resolution geometric encoder-decoder architecture: Based on the set of spatial nodes in the defect-sensitive spatiotemporal segment, construct a multi-resolution point cloud representation. The encoder uses a hierarchical graph convolutional network to extract multi-scale geometric features, and the decoder uses a coordinate-based implicit neural network to map the geometric features and query point coordinates into signed distance function values. Embedded physical information constraints: During the forward propagation of the decoder, the embedded linear elasticity control equations are solved synchronously. The equations take the reconstructed geometry as the computational domain and material property parameters and boundary conditions as inputs to calculate the predicted values ​​of the physical field. A composite loss function comprising data fitting loss, physical field consistency loss, and geometric regularization loss is defined for collaborative optimization. The data fitting loss calculates the difference between the signed distance function of the reconstructed surface and the anomalous observations extracted from the defect-sensitive spatiotemporal segment. The physical field consistency loss calculates the residual between the physical field predicted by the decoder and the standard physical field obtained by numerically solving the governing equations. The geometric regularization loss constrains the curvature and smoothness of the reconstructed surface to ensure geometric rationality. Alternating optimization training is performed, employing a two-stage training strategy. In the first stage, forward data fitting is used to quickly generate a preliminary geometric shape of the defect. In the second stage, the geometric encoder parameters are fixed, and the physical field consistency loss is backpropagated through the physical information neural network layer to iteratively optimize the decoder parameters. This ensures that the reconstructed three-dimensional defect structure satisfies the observation data, while its induced physical field response conforms to the laws of material mechanics.

8. The method according to claim 7, characterized in that, The specific implementation of embedded physical information constraints includes: Adaptive coupling modeling of physical fields: The implicit geometric representation generated by the decoder is dynamically transformed into a computational grid for solving physical fields. This process adopts an adaptive grid refinement strategy, which automatically refines the grid in areas with significant geometric features and uses a sparse grid in flat areas to achieve a balance between computational accuracy and efficiency. Spatial heterogeneity characterization of material parameters: In solving the linear elasticity control equation, the material property parameters are modeled as continuous functions of spatial coordinates rather than uniform constants. A parallel branch of the decoder is trained to output a spatially varying material parameter field, which serves as the input for solving physical constraints to characterize the degradation of material properties caused by concrete aging or steel corrosion. Uncertainty Quantification and Reliable Reconstruction: In the calculation of physical field consistency loss, an uncertainty estimation module based on Bayesian neural network is introduced. This module simultaneously predicts the mean and variance of the physical field residuals and adds an uncertainty-aware loss term to the composite loss function. This guides the model to rely more on data fitting in regions with high physical constraint uncertainty and to strictly follow physical laws in regions with low uncertainty, thereby achieving a reliable reconstruction result with known risks.

9. The method according to claim 1, characterized in that, The multimodal inspection data also includes geographic information data and meteorological data. The method further includes adaptively adjusting the model parameters of the sequence extraction module or the decoder based on the geographic information data and meteorological data through a meta-learning framework to improve the recognition performance under different environments.

10. A multi-source data fusion system for identifying and quantitatively assessing defects in water conservancy projects, used to perform the method as described in any one of claims 1 to 9, characterized in that, The system includes: The data acquisition module is used to acquire multimodal inspection data, including visible light images, infrared thermal imaging data, laser 3D point cloud data, and acoustic echo signals. The topology feature extraction module is used to construct a topology graph structure from the laser 3D point cloud data and to extract spatial topology features that characterize the geometric shape and connection relationship of the engineering structure using a spatiotemporal graph convolutional network. The multimodal fusion module is used to perform cross-modal fusion of the time-frequency features extracted from the acoustic echo signal by wavelet packet energy entropy with the visual features extracted from the visible light image and infrared thermal imaging data, based on the spatial topological structure features, to obtain multimodal fused features. The spatiotemporal segment extraction module is used to model the multimodal fusion features using a sequence extraction module based on the Transformer architecture, and extract defect-sensitive spatiotemporal segments containing spatial location and time interval information; The three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the defect based on the defect-sensitive spatiotemporal segment using a decoder that integrates physical information neural networks to obtain a three-dimensional structural model of the defect. The decoder is configured to simultaneously satisfy the constraints of observation data and the constraints of physical laws characterized by the partial differential equations of mechanics of materials during the reconstruction process. The analysis output module is used to determine the defect type and perform quantitative analysis of the engineering status based on the defect-sensitive spatiotemporal segment and the defect three-dimensional structural model, and output the defect type identification result, the defect three-dimensional structural model and the quantitative evaluation report.

Images