Bridge digital twin disease mapping prediction method and system

Abstract

The application discloses a bridge digital twin disease mapping prediction method and system. The method first acquires multi-angle two-dimensional high-definition images and laser point clouds of the bridge, generates a watertight three-dimensional grid model through cascade registration and fusion, secondly, constructs a hierarchical bounding box acceleration structure based on a surface area heuristic strategy, accurately maps two-dimensional disease pixels to the surface of the three-dimensional model by using a Moller-Trumbore ray tracing algorithm, finally, converts the three-dimensional grid model after mapping the disease into a graph structure, constructs a space-time graph neural network composed of a graph convolution layer and a bidirectional long short-term memory network in cascade, extracts spatial correlation features and time sequence dependence features of the disease, optimizes training through a weighted focal loss function, and outputs a risk heat map of the disease. The application solves the technical problems of disconnection between two-dimensional and three-dimensional spaces, low index efficiency of massive data and lack of dynamic prediction in traditional detection, and realizes fine and intelligent management of the whole life cycle of the bridge.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of civil engineering structural health monitoring, computer graphics and artificial intelligence, and specifically to a method and system for predicting bridge digital twin defects. Background Technology

[0002] The operation and maintenance safety of large bridges, such as long-span suspension bridges and cable-stayed bridges, has attracted much attention. Traditional manual inspection methods suffer from large blind spots, high risk factors, and low efficiency, making it difficult to meet the ever-increasing demands of bridge operation and maintenance. In recent years, drone-based inspection technology combined with computer vision has gradually become more widespread, providing a new technical means for bridge defect detection. However, existing bridge defect detection and management technologies still face the following technical challenges in engineering applications: First, there is a spatial disconnect between two-dimensional images and three-dimensional physical entities, leading to unclear damage location. Damage photos taken by drones are typically two-dimensional images, lacking global spatial coordinate information. Maintenance personnel struggle to accurately correlate local cracks in massive amounts of photos with actual bridge components such as the location of suspension cables or the bottom plate area of ​​stiffening girders, making it impossible to perform targeted repairs and long-term tracking of damage. This technical problem essentially stems from the dimensional gap between two-dimensional visual data and three-dimensional physical space, and existing technologies lack an effective spatial mapping mechanism.

[0003] Secondly, the massive point cloud and mesh data create rendering and computational bottlenecks, making it difficult to meet real-time interactive requirements. Digital twin systems typically need to load hundreds of millions of point clouds or millions of triangular faces, placing extremely high demands on computing resources. Traditional spatial indexing structures, such as octrees or midpoint-partitioned hierarchical bounding boxes, tend to generate a large number of redundant blank nodes when dealing with complex, non-uniformly distributed bridge components, resulting in extremely low efficiency in ray picking and collision detection. When engineers attempt to rotate, scale, or select defective areas in the digital twin model, the system response latency is severe, significantly impacting the usability of engineering applications.

[0004] Furthermore, existing prediction models lack spatial topological correlations, making it impossible to predict spatiotemporal evolution. Current disease prediction methods often treat individual measuring points in isolation, frequently employing single-point time series analysis such as the ARIMA model. This approach ignores the spatial correlation and propagation characteristics of diseases affecting bridges as a whole load-bearing structure. For example, the expansion of crack networks caused by stress concentration at a certain point often propagates along the structural stress path; relying solely on single-point historical data cannot capture this spatial propagation pattern, leading to significant discrepancies between predicted results and actual disease evolution.

[0005] In summary, how to achieve accurate mapping from two-dimensional images to three-dimensional models within the framework of digital twins, construct an efficient spatial index structure to support real-time interaction, and establish a disease evolution prediction model that integrates spatial topological associations have become urgent technical challenges to be solved in this field. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies, such as the spatial disconnect between two-dimensional images and three-dimensional models, low efficiency of massive data indexing, and lack of spatiotemporal evolution prediction, and to provide a method and system for predicting bridge digital twin disease mapping.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: On the one hand, a method for predicting bridge digital twin defects is provided, including the following steps: S1. Acquire multi-angle two-dimensional high-definition images and laser point clouds of the bridge. Align the laser point cloud and the point cloud generated based on the multi-angle two-dimensional high-definition images to the same coordinate system through cascaded registration to obtain a fused point cloud. Perform downsampling and statistical filtering on the fused point cloud to remove outliers and noise. Estimate the normal and set a high-altitude virtual viewpoint for global orientation. Finally, generate a watertight three-dimensional mesh model through Poisson surface reconstruction. S2. Based on the surface area heuristic strategy and combined with the bucket scanning mechanism, a hierarchical bounding box space acceleration structure is constructed for the 3D mesh model; S3. The pixel coordinates of the disease pixels in the multi-angle two-dimensional high-definition image are converted into three-dimensional spatial rays using the camera intrinsic and extrinsic parameter matrix. The intersection points of the three-dimensional spatial rays and the triangular facets in the hierarchical bounding box acceleration structure are calculated using the Möller-Trumbore algorithm. The disease pixels are then mapped to the UV texture coordinates of the three-dimensional mesh model through barycentric coordinate interpolation to obtain the three-dimensional mesh model after mapping the disease. S4. The 3D mesh model after mapping the disease is transformed into a graph structure. A spatiotemporal graph neural network composed of a graph convolutional layer (GCN) and a bidirectional long short-term memory network (Bi-LSTM) is constructed. The graph convolutional layer is used to extract the spatial correlation features of the disease. The spatial correlation features are input into the bidirectional long short-term memory network to extract the temporal dependency features. The forward and reverse hidden states are spliced ​​to form spatiotemporal fusion features. The model is optimized and trained through a weighted focus loss function, and finally the risk heat map of the disease is output.

[0008] Furthermore, S1 specifically includes: The laser point cloud and the point cloud generated based on multi-angle 2D high-definition images are aligned to the same coordinate system through cascaded registration. The fast point feature histogram features of the two sets of point clouds are extracted. The random sampling consensus algorithm is used for global coarse registration, and then the point-to-surface iterative nearest point algorithm is used for local fine registration to obtain the fused point cloud. Voxel downsampling and statistical filtering are performed sequentially on the fused point cloud, and the relationship between each point in the fused point cloud and its corresponding data is calculated. k The average distance between the nearest neighbors Calculate the global average distance mean with standard deviation ,when These were identified as outliers and removed. The threshold coefficient is preset based on the distribution characteristics of the point cloud; A hybrid KD-Tree search strategy is used to determine the local covariance matrix. Principal component analysis is used to extract the eigenvectors corresponding to the minimum eigenvalues ​​as normals. A virtual viewpoint at high altitude is set to force the normals to be uniformly oriented. The point cloud after normal orientation is reconstructed using a Poisson surface to generate a watertight 3D mesh model, and artifacts are clipped according to the sampling density mask.

[0009] Furthermore, S2 specifically includes: Define the surface area heuristic cost function : ; in, The overhead of traversing internal nodes. Find the intersection cost of the triangle. , These are the surface areas of the bounding boxes of the left and right child nodes after the segmentation. , These represent the number of triangles contained in the left and right child nodes, respectively. The surface area of ​​the bounding box of the node before segmentation; A node is a tree-like structural unit organized according to the bounding box spatial position when recursively dividing the triangular facets of a 3D mesh model. It includes leaf nodes that store the actual triangular facets and internal nodes that contain bounding box information of child nodes. The triangular facets are uniformly divided into multiple candidate buckets along the coordinate axes. Only the cost function value at the bucket boundary is evaluated. The partitioning scheme with the minimum cost function value is selected to recursively generate child nodes, thus controlling the time complexity to within a certain range. ,in, N This represents the total number of triangular faces in the 3D mesh model.

[0010] Furthermore, S3 specifically includes: Based on the camera intrinsic and extrinsic parameters, the pixel coordinates are converted into ray origins in the world coordinate system through inverse projection transformation. and direction vector ; For triangular facets in a hierarchical bounding box acceleration structure According to the ray origin and direction vector Determine the ray equation Simultaneous equations of ray and triangle parametric equations Construct a linear system and use Cramer's rule to solve for the distance between the intersection points. and the centroid coordinates of the intersection points And view-side backface culling based on normal dot product is performed during the solution process; Based on the triangular facet where the intersection point is located Texture coordinates corresponding to the three vertices and the centroid coordinates of the intersection points ,in accordance with Calculate the three-dimensional texture coordinates to obtain the three-dimensional mesh model after mapping the disease.

[0011] Furthermore, in S4, the 3D mesh model after mapping the disease is transformed into a graph structure. Specifically, this includes: using the vertices of the 3D mesh model as graph nodes of the graph structure, constructing an adjacency matrix based on the spatial Euclidean distance and radial basis function between nodes, and when the distance between nodes is less than a set threshold, the elements of the adjacency matrix are radial basis function values, otherwise they are 0.

[0012] Furthermore, in S4, graph convolutional layers are used to extract spatial correlation features of diseases. Specifically, this includes: using spectral domain graph convolutional networks for message passing, and using cascaded graph convolutional layers to aggregate second-order topological neighborhood features of nodes to extract spatial correlation features of diseases and perceive the spread trend of crack networks.

[0013] Furthermore, in S4, the spatial correlation features extracted by the graph convolutional layer are input into the bidirectional long short-term memory network unit, which captures the cumulative effect of fatigue damage in the positive direction and provides contextual constraints in the negative direction. The positive and negative hidden states are then concatenated to output the spatiotemporal fusion features.

[0014] Furthermore, in S4, model optimization training is performed using a weighted focus loss function, specifically including: employing a weighted focus loss function. As training loss, among To focus parameters, To dynamically adjust class weights based on the ratio of positive to negative samples, The model predicts the probability of the true class; preferably, the model optimization training uses the AdamW optimizer in conjunction with a cosine annealing learning rate for parameter updates.

[0015] Furthermore, S4 outputs a risk heat map of the disease, specifically including: generating a risk heat map on the surface of a three-dimensional mesh model based on the spatiotemporal fusion features output by the spatiotemporal graph neural network, dynamically displaying the spatiotemporal evolution trend of the disease.

[0016] On the other hand, a system for implementing the above method is provided, comprising: The data acquisition and preprocessing module is used to acquire multi-angle two-dimensional high-definition images and laser point clouds of the bridge. The laser point cloud and the point cloud generated based on the multi-angle two-dimensional high-definition images are aligned to the same coordinate system through cascaded registration to obtain a fused point cloud. Downsampling and statistical filtering are performed on the fused point cloud to remove outliers and noise. The normal is estimated and a high-altitude virtual viewpoint is set for global orientation. Finally, a watertight three-dimensional mesh model is generated by reconstructing the Poisson surface. The spatial indexing and mapping module is used to construct a hierarchical bounding box spatial acceleration structure for the 3D mesh model based on a surface area heuristic strategy combined with a bucket scanning mechanism. It uses the camera intrinsic and extrinsic parameter matrix to convert the pixel coordinates of disease pixels in multi-angle 2D high-definition images into 3D spatial rays. It uses the Möller-Trumbore algorithm to calculate the intersection points of the 3D spatial rays and the triangular facets in the hierarchical bounding box acceleration structure. Finally, it uses barycentric coordinate interpolation to map the disease pixels to the UV texture coordinates of the 3D mesh model, thus obtaining the 3D mesh model after mapping the disease. The intelligent evolution prediction module is used to transform the 3D mesh model after mapping the disease into a graph structure, construct a spatiotemporal graph neural network composed of graph convolutional layers and bidirectional long short-term memory networks, use graph convolutional layers to extract spatial correlation features of the disease, input the spatial correlation features into the bidirectional long short-term memory network to extract temporal dependency features, and splice the forward and reverse hidden states to form spatiotemporal fusion features. The model is optimized and trained through a weighted focus loss function, and finally outputs a risk heat map of the disease. The digital twin visualization module is used to render a 3D mesh model and the mapped disease texture, and dynamically display a risk heat map.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention uses coordinate system transformation and Möller-Trumbore centroid interpolation to accurately project two-dimensional disease images onto the surface of a three-dimensional mesh model, achieving sub-pixel-level texture mapping with an average positioning error of less than 2mm. This solves the technical problem of the disconnect between two-dimensional images and three-dimensional physical entities in traditional detection.

[0018] 2. This invention constructs a hierarchical bounding box space acceleration structure based on a surface area heuristic strategy, and combines a bucketing scanning mechanism with backface culling optimization to control the time complexity of ray intersection in scenarios with tens of millions of faces to within a certain range. The query efficiency is improved by more than 50% compared with the traditional octree, which meets the real-time interaction needs of massive point cloud and grid data.

[0019] 3. This invention transforms a three-dimensional mesh model into a graph structure and constructs a spatiotemporal graph neural network composed of a graph convolutional layer (GCN) and a bidirectional long short-term memory network (Bi-LSTM). The graph convolutional layer aggregates second-order topological neighborhood features to extract spatial correlations, and the bidirectional long short-term memory network captures temporal dependencies to form spatiotemporal fusion features. Combined with a weighted focus loss function, it solves the problem of extreme imbalance in disease samples, enabling the prediction accuracy of major diseases to reach over 90%, thus realizing a leap from post-discovery to pre-discovery warning.

[0020] 4. This invention forms a complete automated closed loop from multi-source data acquisition, cascaded registration, Poisson reconstruction, to SAH-BVH (surface area heuristic boundary volume hierarchy) construction, ray tracing mapping, and then to spatiotemporal graph neural network training and risk heat map output, without the need for manual intervention, which greatly reduces the labor cost and operational risks of bridge inspection. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method of the present invention.

[0022] Figure 2 This is a performance comparison chart between the present invention and the traditional brute-force traversal.

[0023] Figure 3 This is a risk heat map of the spatiotemporal evolution of bridge defects output by an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] like Figure 1 As shown, the present invention provides a bridge digital twin disease mapping prediction method, which includes the following steps: S1. Acquire multi-angle two-dimensional high-definition images and laser point clouds of the bridge. Align the laser point cloud and the point cloud generated based on the multi-angle two-dimensional high-definition images to the same coordinate system through cascaded registration to obtain a fused point cloud. Perform downsampling and statistical filtering on the fused point cloud to remove outliers and noise. Estimate the normal and set a high-altitude virtual viewpoint for global orientation. Finally, generate a watertight three-dimensional mesh model through Poisson surface reconstruction. S2. Based on the surface area heuristic strategy and combined with the bucket scanning mechanism, a hierarchical bounding box space acceleration structure is constructed for the 3D mesh model; S3. The pixel coordinates of the disease pixels in the multi-angle two-dimensional high-definition image are converted into three-dimensional spatial rays using the camera intrinsic and extrinsic parameter matrix. The intersection points of the three-dimensional spatial rays and the triangular facets in the hierarchical bounding box acceleration structure are calculated using the Möller-Trumbore algorithm. The disease pixels are then mapped to the UV texture coordinates of the three-dimensional mesh model through barycentric coordinate interpolation to obtain the three-dimensional mesh model after mapping the disease. S4. The 3D mesh model after mapping the disease is transformed into a graph structure. A spatiotemporal graph neural network composed of graph convolutional layers and bidirectional long short-term memory networks is constructed. The graph convolutional layers are used to extract the spatial correlation features of the disease. The spatial correlation features are input into the bidirectional long short-term memory network to extract the temporal dependency features. The forward and reverse hidden states are spliced ​​to form spatiotemporal fusion features. The model is optimized and trained through a weighted focus loss function, and finally, a risk heat map of the disease is output.

[0026] This invention integrates multi-angle two-dimensional high-definition images with laser point clouds to construct a high-precision three-dimensional mesh model. It utilizes SAH-BVH accelerated structures and ray tracing to achieve sub-pixel-level accurate mapping of diseases. Furthermore, it uses spatiotemporal graph neural networks to extract the spatial correlation and temporal evolution features of diseases, forming a fully automated closed loop from three-dimensional reconstruction and disease mapping to intelligent prediction. This effectively solves the technical problems of fuzzy positioning, low indexing efficiency, and lack of dynamic prediction in traditional detection.

[0027] In one specific embodiment of the present invention, an industrial-grade drone equipped with a high-resolution full-frame camera and an airborne LiDAR is used to perform a full-coverage scan of the bridge, acquiring multi-angle two-dimensional high-definition images and laser point clouds of the bridge.

[0028] In one specific embodiment of the present invention, S1 specifically includes: Heterogeneous Point Cloud Cascade Registration: While laser point clouds offer high precision but lack texture, point clouds generated from multi-angle 2D high-resolution images contain rich RGB colors but are prone to geometric distortion. This invention aligns laser point clouds and point clouds generated from multi-angle 2D high-resolution images to the same coordinate system through cascade registration. First, fast point feature histograms are extracted from both sets of point clouds, and global coarse registration is performed using the Random Sample Consensus (RANSAC) algorithm to eliminate initial pose deviations. Then, local fine registration is performed using the Iterative Closest Point (ICP) algorithm, using the distance from the point to the target surface as the error metric, iteratively optimizing the transformation matrix to obtain a fused point cloud.

[0029] Point cloud filtering optimization: Voxel downsampling and statistical filtering are performed sequentially on the fused point cloud. Voxel downsampling is used to uniformly sparsify the point cloud and reduce the amount of data; statistical filtering is used to remove outliers and noise, calculating the relationship between each point and its corresponding data. k The average distance between the nearest neighbors Calculate the global average distance mean with standard deviation ,when Points were identified as outliers and removed. The threshold coefficient is preset based on the distribution characteristics of the point cloud.

[0030] Line estimation and global orientation: A hybrid KD-Tree search strategy is used to determine the local covariance matrix. Principal component analysis (PCA) is used to extract the eigenvectors corresponding to the minimum eigenvalues ​​as normals. A virtual viewpoint at high altitude is set to force the normals to be uniformly oriented, thus eliminating topology flip errors.

[0031] Poisson Surface Reconstruction: A watertight 3D mesh model is generated from the point cloud after normal orientation using Poisson surface reconstruction, and artifacts are clipped according to the sampling density mask. Specifically, the point cloud after normal orientation is converted into a spatial normal vector field, and the Poisson equation of the indicator function is solved. Extract isosurfaces to generate a watertight 3D mesh model.

[0032] In a preferred embodiment of the present invention, the threshold in the point cloud filtering optimization step... The values ​​are not randomly selected, but are set based on the physical distribution characteristics of the bridge point cloud and the environmental noise level. The specific setting methods include the following implementation forms, which can be flexibly selected according to actual computing resources and accuracy requirements.

[0033] Implementation Mode 1: Preset Constant Mode (Applicable to general fast processing scenarios).

[0034] In general bridge inspection scenarios, in order to balance filtering efficiency and feature preservation integrity, a preset constant is used, with an effective value range of 1.0-3.0.

[0035] Preferred value: =2.0.

[0036] like If the value is too small, the filtering is too strict, and it is very easy to misidentify the edge features of the bridge (such as the sharp angles of the steel box girder, small cables, etc.) as outliers and remove them, resulting in the loss of key structural information; if If the value is too large, the filter will be too loose, failing to effectively remove outlier noise caused by airborne dust, bird activity, or multipath effects, thus affecting the point cloud quality. Verification based on extensive bridge measurement data shows that... At a resolution of 2.0, it can maintain the structural integrity of the bridge to the greatest extent while ensuring a noise filtering rate of over 85%.

[0037] Implementation Mode 2: Adaptive Calculation Mode (suitable for high-precision analysis scenarios).

[0038] For complex bridge environments, if a fixed threshold coefficient is used... It is difficult to simultaneously satisfy the requirements of high precision for dense point clouds and high completeness for sparse point clouds. This invention provides an adaptive determination method based on point cloud distribution characteristics. The specific calculation formula is as follows: ; in: The baseline threshold coefficient has a value range of [1.0, 3.0], preferably 2.0; The average point cloud density of the current local area (unit: points / m) 2 ); This is a preset reference point cloud density threshold based on the scanning equipment parameters and measurement distance; The real-time noise level is estimated based on the sensor signal-to-noise ratio of the current scan data; This represents the reference noise level under standard environmental conditions. , These are the point cloud density weighting factor and the noise level weighting factor, respectively, with values ​​ranging from [0.2, 0.8] to [0.1, 0.5], and the preferred values ​​being 0.5 and 0.3, respectively.

[0039] Using the above formula, when the point cloud density decreases (e.g., during remote scanning), Decrease Increase, the system will automatically adjust the height. Values ​​are used to prevent sparse structural points from being misidentified as noise points; when the noise level increases, Increase, the system automatically decreases This enhances noise reduction capabilities. The mechanism achieves an adaptive filtering effect, rigorously filtering where necessary and strictly preserving where appropriate, effectively filtering out environmental noise while maintaining the integrity of structural features.

[0040] Implementation Mode 3: Segmented assignment mode based on bridge type experience database (applicable to bridge type detection).

[0041] To balance processing efficiency with the different materials and structural characteristics of bridges, this invention provides statistical filtering threshold coefficients for different types of bridges. The method for assigning experience values ​​is as follows: Steel box girder bridge: The value range is [1.5, 2.5], preferably 2.0. The surface of the steel box girder has good laser reflection characteristics, high point cloud density and low noise, so a relatively small threshold can be used for accurate filtering, effectively preserving the structural edge features.

[0042] Concrete bridge: The value range is [2.0, 3.0], preferably 2.5. Concrete surfaces are relatively rough and prone to multipath scattering noise. Appropriately relaxing the threshold can prevent misclassification of true structural points on rough surfaces as outliers, ensuring the integrity of the concrete surface texture.

[0043] Suspension bridges / cable-stayed bridges: The value range is [2.5, 3.5], preferably 3.0. This type of bridge contains a large number of slender components such as cables and suspension cables, and its local point cloud exhibits discrete distribution characteristics. A larger threshold must be used to prevent the point cloud of key structures such as cables from being excessively removed as outliers, thus ensuring the geometric continuity of the slender components.

[0044] In one specific embodiment of the present invention, S2 specifically includes: SAH Cost Function Definition: To avoid the performance disaster caused by exhaustive search, the surface area heuristic (SAH) strategy is used to define the cost function. For the current node's bounding box, the surface area heuristic cost function is defined. for: ;

[0045] in The overhead of traversing internal nodes. Find the intersection cost of the triangle. , These are the surface areas of the bounding boxes of the left and right child nodes after the segmentation. , These represent the number of triangles contained in the left and right child nodes, respectively. The surface area of ​​the bounding box of the node before segmentation; Bucketing optimization: The triangular facet is evenly divided into multiple candidate buckets along the coordinate axes. Only the cost function value at the bucket boundary is evaluated. The segmentation scheme with the minimum cost function value is selected to recursively generate child nodes, thus controlling the time complexity to within a certain range. ,in, N This represents the total number of triangular faces in the 3D mesh model.

[0046] A node is a tree-like structural unit organized according to the bounding box spatial position when recursively partitioning triangular faces of a 3D mesh model. It includes leaf nodes that store the actual triangular faces and internal nodes that contain bounding box information of child nodes. The recursion terminates when the number of triangular faces contained in the current node is less than or equal to a preset threshold, or when the current node reaches a preset maximum tree depth.

[0047] In one specific embodiment of the present invention, S3 specifically includes: Pixel coordinates to ray conversion: This method uses the camera's intrinsic and extrinsic parameter matrices to convert the pixel coordinates of defective pixels in multi-angle 2D high-resolution images into 3D spatial rays. Let the pixel coordinates of the defective pixels be... The camera intrinsic parameter matrix is K The camera extrinsic matrix is The relationship between pixel coordinates and points in three-dimensional space satisfies: ; in, This represents the depth value in the camera coordinate system. These are the world coordinates of a point in three-dimensional space.

[0048] By performing a matrix inversion transformation, the starting point of the ray in the world coordinate system can be obtained. and direction vector . Specifically: Ray origin This refers to the world coordinates of the UAV camera's optical center at the current moment of image acquisition, determined by the camera's extrinsic parameter matrix. The translation vector t is determined; the ray direction vector is determined. It is the unit direction vector pointing from the camera's optical center to the corresponding spatial point of the pixel, which can be calculated using the inverse projection formula.

[0049] Möller-Trumbore ray intersection: for triangular faces in a hierarchical bounding box accelerated structure According to the ray origin and direction vector Determine the ray equation Simultaneous equations of ray and triangle parametric equations ,get: ; By rearranging and simplifying, we obtain the linear system: ; By applying Cramer's rule to solve the above linear system, the intersection distance λ in the ray equation and the centroid coordinates (u,v) of the intersection points in the triangular parametric equation are obtained. This avoids the division-by-zero error caused by calculating the plane normal, significantly improving the numerical stability during parallel computation. Furthermore, backface culling based on the dot product of the normals is performed during the solution process. If the dot product of the ray direction and the normal of the triangular facet is greater than 0, it is considered a backface and the intersection calculation for that facet is skipped.

[0050] Centroid coordinate interpolation mapping: based on the triangular facet where the intersection point is located Texture coordinates corresponding to the three vertices and the centroid coordinates of the intersection points The three-dimensional texture coordinates are calculated according to the following formula: ; Through the above interpolation calculation, the disease pixels in the two-dimensional image are accurately mapped to the surface of the three-dimensional mesh model, and the three-dimensional mesh model after mapping the disease is obtained.

[0051] Multi-view visibility detection: Since the same defect is often covered by multiple photos when collected by drones, this invention also introduces a multi-view visibility detection mechanism. When the same triangular facet is covered by multiple images, the angle between the ray corresponding to each image and the normal of the facet is calculated. Only the pixel texture corresponding to the ray with the smallest angle and not blocked by other structures is accepted, thus eliminating texture ghosting and blurring caused by multi-angle projection.

[0052] In a preferred embodiment, the present invention employs a multi-process parallel computing architecture, distributing the intersection calculation task of three-dimensional spatial rays to multiple processing cores for simultaneous execution. Specifically, the three-dimensional spatial rays corresponding to the lesion pixels in multi-angle two-dimensional high-definition images are evenly distributed to multiple processes. Each process independently completes the intersection calculation of the rays with the triangular facets in the hierarchical bounding box acceleration structure, and maps the lesion pixels to the UV texture coordinates of the three-dimensional mesh model through barycentric coordinate interpolation. Parallel computing fully utilizes the computing power of multi-core processors, significantly shortening the processing time of massive image data. In a specific embodiment of the present invention, S4 specifically includes: Graph Structure Construction: The vertices of the mapped 3D mesh model are used as graph nodes. An adjacency matrix A is constructed based on the spatial Euclidean distance between nodes. The adjacency matrix A is constructed using a thresholded radial basis function, and the element Aij of the adjacency matrix A is defined as follows: ; Where xi and xj are the coordinates of the graph nodes in three-dimensional space; |xi−xj| 2Let be the spatial Euclidean distance between nodes, r be a set threshold, and σ' be the bandwidth parameter of the radial basis function. Using the above construction method, the 3D mesh model is transformed into a graph structure G = (V, E, A).

[0053] Graph Convolutional Layer Spatial Aggregation: A spectral domain graph convolutional network is used for message passing to extract spatial correlation features of diseases. Based on the constructed adjacency matrix A, self-connections are added to preserve the individual features of each node, resulting in: , in, It is an identity matrix.

[0054] right After normalization, the propagation rule of the graph convolutional layer is as follows: ; in, for The degree matrix, For the first The node feature matrix of the layer For the first The trainable weight matrix of the layer, The activation function is denoted as . By cascading multiple graph convolutional layers, the nodes aggregate second-order topological neighborhood features, extracting spatial correlation features of the disease and perceiving the spread trend of the crack network.

[0055] Temporal modeling of bidirectional long short-term memory (LSTM) networks: Spatial correlation features extracted by graph convolutional layers are arranged in chronological order to form a spatial feature sequence, which is then input into the bidirectional LSTM network units. The bidirectional LSTM network consists of a forward LSTM and a backward LSTM: the forward LSTM captures the cumulative effect of fatigue damage. The system extracts positive patterns of disease evolution from historical data; a reverse LSTM provides contextual constraints to extract inverse dependencies from future information. The forward and reverse hidden states are concatenated to output a spatiotemporal fusion feature. : ; in, This is a positive hidden state. This is the reverse hidden state. This indicates a vector concatenation operation.

[0056] Model Optimization Training: Since the defective areas (such as cracks) on a bridge constitute a very small percentage (usually less than 1%) of the entire massive bridge surface, this represents a typical problem of extremely imbalanced sample distribution. If conventional cross-entropy loss is used, the model will tend to predict all defects as absent. This invention introduces a weighted focal loss function during the training phase. The weighted focal loss function is defined as follows: ; in, To focus on parameters, the model is forced to reduce the weight of easily distinguishable samples, making it focus on difficult samples; To dynamically adjust class weights based on the ratio of positive to negative samples, This represents the model's predicted probability for the true class. Simultaneously, the AdamW optimizer, combined with a cosine annealing learning rate, is used for parameter updates to effectively prevent overfitting.

[0057] Output risk heat map: Based on the spatiotemporal fusion features output by the spatiotemporal graph neural network, a risk heat map is generated on the surface of the three-dimensional mesh model, dynamically displaying the spatiotemporal evolution trend of the disease, with high-risk areas marked in red.

[0058] The present invention also provides a system for implementing the above method, comprising: The data acquisition and preprocessing module is used to acquire multi-angle two-dimensional high-definition images and laser point clouds of the bridge. The laser point cloud and the point cloud generated based on the multi-angle two-dimensional high-definition images are aligned to the same coordinate system through cascaded registration to obtain a fused point cloud. Downsampling and statistical filtering are performed on the fused point cloud to remove outliers and noise. The normal is estimated and a high-altitude virtual viewpoint is set for global orientation. Finally, a watertight three-dimensional mesh model is generated by reconstructing the Poisson surface. The spatial indexing and mapping module is used to construct a hierarchical bounding box spatial acceleration structure for the 3D mesh model based on a surface area heuristic strategy combined with a bucket scanning mechanism. It uses the camera intrinsic and extrinsic parameter matrix to convert the pixel coordinates of disease pixels in multi-angle 2D high-definition images into 3D spatial rays. It uses the Möller-Trumbore algorithm to calculate the intersection points of the 3D spatial rays and the triangular facets in the hierarchical bounding box acceleration structure. Finally, it uses barycentric coordinate interpolation to map the disease pixels to the UV texture coordinates of the 3D mesh model, thus obtaining the 3D mesh model after mapping the disease. The intelligent evolution prediction module is used to transform the 3D mesh model after mapping the disease into a graph structure, construct a spatiotemporal graph neural network composed of graph convolutional layers and bidirectional long short-term memory networks, use graph convolutional layers to extract spatial correlation features of the disease, input the spatial correlation features into the bidirectional long short-term memory network to extract temporal dependency features, and splice the forward and reverse hidden states to form spatiotemporal fusion features. The model is optimized and trained through a weighted focus loss function, and finally outputs a risk heat map of the disease. The digital twin visualization module is used to render a 3D mesh model and the mapped disease texture, and dynamically display a risk heat map.

[0059] This invention demonstrated excellent performance in basic performance tests. For example... Figure 2 As shown in Table 1, the present invention (SAH-BVH, i.e., BVH acceleration algorithm) has a significant advantage in the efficiency of high-frequency ray query, improving it by more than 50% compared with traditional methods, thus breaking the rendering bottleneck.

[0060] Table 1 Comparison of Accelerated Structural Performance

[0061] Table 1 compares the performance of three BVH construction strategies in scenarios with tens of millions of triangles. The SAH-BVH strategy used in this invention has a construction time of 1240ms and a ray query efficiency of 28.6 MRays / s, which is about 128% higher than LBVH and about 57% higher than midpoint segmentation BVH. Although the construction time is slightly longer, it significantly improves the ray query efficiency, breaks through the rendering interaction bottleneck of digital twin systems, and has important application value for bridge defect mapping scenarios that require high-frequency ray intersection.

[0062] To more intuitively illustrate the practical application effect of the present invention, this embodiment takes the digital twin monitoring project of the steel box girder and cable system of a cross-river super long span suspension bridge (main span 1080 meters) as an example to conduct a full process review.

[0063] This suspension bridge has been in service for over 15 years, and the bottom of the steel box girder is prone to fatigue cracks under heavy traffic. Previously, manual inspections were time-consuming and lacked precise location accuracy. This embodiment utilizes an industrial-grade drone equipped with a high-resolution camera and LiDAR to complete a full scan of the bridge's underside over three days. Input data volume: 55,000 high-resolution surface images and approximately 120 million raw LiDAR point cloud coordinates were acquired, with a total raw data volume of 80GB.

[0064] After importing the massive amount of data into the computing server containing this system, the system automatically executes the following pipeline: After loading the original point cloud, voxel downsampling is performed, leaving approximately 35 million points. Subsequently, statistical filtering is performed to remove outliers and noise. Finally, a 3D mesh model with approximately 8.45 million triangular faces is successfully generated through Poisson reconstruction, and the watertightness verification is passed. A BVH acceleration structure is constructed based on the SAH bucketing strategy, with a tree depth of 28 layers and a total of approximately 16.9 million nodes. Multi-process Möller-Trumbore ray tracing mapping is initiated, and ray intersection and texture interpolation calculations are completed for 55,000 images, accurately mapping the 2D disease texture to 3D faces, with a global average backprojection positioning error of 1.8 mm. The ST-GNN spatiotemporal graph neural network model was loaded, and a topological graph adjacency matrix was constructed with 8.45 million nodes and 25.3 million edges. Combined with 36 months of historical inspection data, bidirectional LSTM inference was initiated to predict the disease evolution trend in the next 6 months. Finally, a future risk prediction heat map was generated, and 3 high-risk crack spread areas were found, with the warning level set to red.

[0065] After S3 ray transformation and barycentric coordinate interpolation mapping, 55,000 originally discrete photos were precisely fitted onto the three-dimensional steel box girder model. Engineers can freely rotate and scale the bridge in the three-dimensional rendering engine and clearly see the spatial three-dimensional coordinates of each crack (accurate to the millimeter).

[0066] Figure 3 This is a risk heatmap of the spatiotemporal evolution of bridge defects output by an embodiment of the present invention. In the figure, the color intensity represents the predicted crack width / risk index, and the red area is the high-risk warning area. Figure 3 The data shows the spread trend and spatial distribution of the cracks within future time windows. Time t=0 months represents the current monitoring, time t=3 months represents the short-term forecast, and time t=6 months represents the high-risk warning.

[0067] Subsequent engineering verification showed that after receiving the early warning, the bridge maintenance center dispatched technicians with ultrasonic flaw detectors to conduct a special review at the predicted locations. The results confirmed that the internal steel structures in these three areas had already shown early fatigue yielding, which was consistent with the prediction model. Based on the three-dimensional coordinates provided by the system, the maintenance team directly reinforced the structures with carbon fiber plates, successfully avoiding potential structural failure.

[0068] Finally, it should be noted that the above embodiments are merely preferred embodiments of the present invention used to illustrate the technical solutions of the present invention, and are not intended to limit the invention, nor are they intended to limit the patent scope of the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention. That is to say, any changes or refinements made to the main design concept and spirit of the present invention that are not of substantial significance, but whose technical problems are still consistent with the present invention, should be included within the protection scope of the present invention. In addition, the direct or indirect application of the technical solutions of the present invention to other related technical fields are similarly included within the patent protection scope of the present invention.

Claims

1. A method for predicting bridge digital twin defects, characterized in that, Includes the following steps: S1. Acquire multi-angle two-dimensional high-definition images and laser point clouds of the bridge. Align the laser point cloud and the point cloud generated based on the multi-angle two-dimensional high-definition images to the same coordinate system through cascaded registration to obtain a fused point cloud. Perform downsampling and statistical filtering on the fused point cloud to remove outliers and noise. Estimate the normal and set a high-altitude virtual viewpoint for global orientation. Finally, generate a watertight three-dimensional mesh model through Poisson surface reconstruction. S2. Based on the surface area heuristic strategy and combined with the bucket scanning mechanism, a hierarchical bounding box space acceleration structure is constructed for the 3D mesh model; S3. The pixel coordinates of the disease pixels in the multi-angle two-dimensional high-definition image are converted into three-dimensional spatial rays using the camera intrinsic and extrinsic parameter matrix. The intersection points of the three-dimensional spatial rays and the triangular facets in the hierarchical bounding box acceleration structure are calculated using the Möller-Trumbore algorithm. The disease pixels are then mapped to the UV texture coordinates of the three-dimensional mesh model through barycentric coordinate interpolation to obtain the three-dimensional mesh model after mapping the disease. S4. The 3D mesh model after mapping the disease is transformed into a graph structure. A spatiotemporal graph neural network composed of graph convolutional layers and bidirectional long short-term memory network is constructed. The graph convolutional layers are used to extract the spatial correlation features of the disease. The spatial correlation features are input into the bidirectional long short-term memory network to extract the temporal dependency features. The forward and reverse hidden states are spliced ​​to form spatiotemporal fusion features. The model is optimized and trained through a weighted focus loss function. Finally, the risk heat map of the disease is output. S1 specifically includes: The laser point cloud and the point cloud generated based on multi-angle 2D high-definition images are aligned to the same coordinate system through cascaded registration. The fast point feature histogram features of the two sets of point clouds are extracted. The random sampling consensus algorithm is used for global coarse registration, and then the point-to-surface iterative nearest point algorithm is used for local fine registration to obtain the fused point cloud. Voxel downsampling and statistical filtering are performed sequentially on the fused point cloud, and the relationship between each point in the fused point cloud and its corresponding data is calculated. k The average distance between the nearest neighbors Calculate the global average distance mean with standard deviation ,when These were identified as outliers and removed. The threshold coefficient is preset based on the distribution characteristics of the point cloud; A hybrid KD-Tree search strategy is used to determine the local covariance matrix. Principal component analysis is used to extract the eigenvectors corresponding to the minimum eigenvalues ​​as normals. A virtual viewpoint at high altitude is set to force the normals to be uniformly oriented. The point cloud after normal orientation is reconstructed using a Poisson surface to generate a watertight 3D mesh model, and artifacts are clipped according to the sampling density mask; S2 specifically includes: Define the surface area heuristic cost function : ； in, The overhead of traversing internal nodes. Find the intersection cost of the triangle. , These are the surface areas of the bounding boxes of the left and right child nodes after the segmentation. , These represent the number of triangles contained in the left and right child nodes, respectively. The surface area of ​​the bounding box of the node before segmentation; A node is a tree-like structural unit organized according to the bounding box spatial position when recursively dividing the triangular facets of a 3D mesh model. It includes leaf nodes that store the actual triangular facets and internal nodes that contain bounding box information of child nodes. The triangular facets are uniformly divided into multiple candidate buckets along the coordinate axes. Only the cost function value at the bucket boundary is evaluated. The partitioning scheme with the minimum cost function value is selected to recursively generate child nodes, thus controlling the time complexity to within a certain range. ,in, N This represents the total number of triangular faces in the 3D mesh model. S3 specifically includes: Based on the camera intrinsic and extrinsic parameters, the pixel coordinates are converted into ray origins in the world coordinate system through inverse projection transformation. and direction vector ; For triangular facets in a hierarchical bounding box acceleration structure According to the ray origin and direction vector Determine the ray equation Simultaneous equations of ray and triangle parametric equations Construct a linear system and use Cramer's rule to solve for the distance between the intersection points. and the centroid coordinates of the intersection points And view-side backface culling based on normal dot product is performed during the solution process; Based on the triangular facet where the intersection point is located Texture coordinates corresponding to the three vertices and the centroid coordinates of the intersection points ,in accordance with Calculate the three-dimensional texture coordinates to obtain the three-dimensional mesh model after mapping the disease; In S4, the 3D mesh model after mapping the disease is transformed into a graph structure. Specifically, the vertices of the 3D mesh model are used as graph nodes of the graph structure. An adjacency matrix is ​​constructed based on the spatial Euclidean distance and radial basis function between nodes. When the distance between nodes is less than a set threshold, the elements of the adjacency matrix are radial basis function values; otherwise, they are 0.

2. The prediction method according to claim 1, characterized in that, S4 utilizes graph convolutional layers to extract spatial correlation features of diseases. Specifically, it employs spectral domain graph convolutional networks for message passing and cascaded graph convolutional layers to aggregate second-order topological neighborhood features of nodes, thereby extracting spatial correlation features of diseases and perceiving the spread trend of crack networks.

3. The prediction method according to claim 1, characterized in that, In S4, the spatial correlation features extracted by the graph convolutional layer are input into the bidirectional long short-term memory network unit. The positive side captures the cumulative effect of fatigue damage, the negative side provides contextual constraints, and the positive and negative hidden states are concatenated to output spatiotemporal fusion features.

4. The prediction method according to claim 1, characterized in that, In S4, model optimization training is performed using a weighted focus loss function, specifically including: employing a weighted focus loss function. As training loss, among To focus parameters, To dynamically adjust class weights based on the ratio of positive to negative samples, The model predicts the probability of the true class; the model optimization training uses the AdamW optimizer with cosine annealing learning rate to update parameters.

5. The prediction method according to claim 1, characterized in that, S4 outputs a risk heat map of the disease, specifically including: generating a risk heat map on the surface of a three-dimensional mesh model based on the spatiotemporal fusion features output by the spatiotemporal graph neural network, dynamically displaying the spatiotemporal evolution trend of the disease.

6. A system for implementing the method according to any one of claims 1 to 5, characterized in that, include: The data acquisition and preprocessing module is used to acquire multi-angle two-dimensional high-definition images and laser point clouds of the bridge. The laser point cloud and the point cloud generated based on the multi-angle two-dimensional high-definition images are aligned to the same coordinate system through cascaded registration to obtain a fused point cloud. Downsampling and statistical filtering are performed on the fused point cloud to remove outliers and noise. The normal is estimated and a high-altitude virtual viewpoint is set for global orientation. Finally, a watertight three-dimensional mesh model is generated by reconstructing the Poisson surface. The spatial indexing and mapping module is used to construct a hierarchical bounding box spatial acceleration structure for the 3D mesh model based on a surface area heuristic strategy combined with a bucket scanning mechanism. It uses the camera intrinsic and extrinsic parameter matrix to convert the pixel coordinates of disease pixels in multi-angle 2D high-definition images into 3D spatial rays. It uses the Möller-Trumbore algorithm to calculate the intersection points of the 3D spatial rays and the triangular facets in the hierarchical bounding box acceleration structure. Finally, it uses barycentric coordinate interpolation to map the disease pixels to the UV texture coordinates of the 3D mesh model, thus obtaining the 3D mesh model after mapping the disease. The intelligent evolution prediction module is used to transform the 3D mesh model after mapping the disease into a graph structure, construct a spatiotemporal graph neural network composed of graph convolutional layers and bidirectional long short-term memory networks, use graph convolutional layers to extract spatial correlation features of the disease, input the spatial correlation features into the bidirectional long short-term memory network to extract temporal dependency features, and splice the forward and reverse hidden states to form spatiotemporal fusion features. The model is optimized and trained through a weighted focus loss function, and finally outputs a risk heat map of the disease. The digital twin visualization module is used to render a 3D mesh model and the mapped disease texture, and dynamically display a risk heat map.

Images