Structural damage detection method and system based on multi-modal fusion

Abstract

The invention relates to the technical field of image processing, in particular to a structural damage detection method and system based on multi-modal fusion, and the method comprises the steps: collecting a two-dimensional image and a three-dimensional point cloud of structural damage; segmenting the two-dimensional image by using a deep learning model to obtain a segmented image; mapping a three-dimensional coordinate of the three-dimensional point cloud to an image coordinate, establishing a corresponding relationship between the three-dimensional point cloud and pixels of the segmented image, and obtaining a damage category label corresponding to the three-dimensional point cloud according to a segmentation result of each pixel in the segmented image; and according to the damage category label of the three-dimensional point cloud, structural data of the crack and the planar damage are extracted according to geometric characteristics of the damage. According to the invention, the accuracy and consistency of damage detection results are ensured, and the precision and robustness of damage detection are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a structural damage detection method and system based on multi-modal fusion. Background Art

[0002] With the rapid development of computer vision, artificial intelligence and sensor technology in the 21st century, the application of structural damage detection technology in infrastructure health monitoring has become particularly important. The following are the main detection technologies in the field of structural damage detection:

[0003] (1) Image-based damage detection alone

[0004] Image-based damage detection technology alone uses feature information in two-dimensional images to identify and classify surface defects. Common methods include traditional image processing techniques (such as edge detection and feature matching) and deep learning models (such as U-Net and Mask R-CNN). This type of technology mainly relies on the high-resolution advantage of images to capture tiny cracks and other detailed features. For example, deep learning methods generate semantic segmentation results of damage by learning image features, which has high detection accuracy and robustness. However, the disadvantage of this method is that it cannot accurately describe the location and range of damage in three-dimensional space, and can only provide two-dimensional distribution information of damage. In addition, for complex structural surfaces or scenes with occlusion, the detection performance will be significantly reduced. The applicability of this method is limited in scenarios where spatial analysis and quantitative evaluation of damage are required.

[0005] (2) Damage detection based on point cloud alone

[0006] Point cloud data is three-dimensional geometric information obtained by laser radar (LiDAR), structured light or other sensors, and is widely used in the field of structural health monitoring. Damage detection technology based on point cloud alone usually uses geometric information, such as surface normal vector, distance change or clustering method, to identify the damaged area. In recent years, deep learning technology has also begun to be applied to the processing of point cloud data, such as PointNet, PointCNN and other models, which can directly classify and segment damage in point cloud space. The main advantage of damage detection based on point cloud is that it can accurately obtain three-dimensional spatial information, which is suitable for evaluating the location, direction, size and other characteristics of damage, especially for complex geometric structures. However, the resolution of point cloud data is usually low, and it is difficult to capture tiny surface details, such as cracks and surface texture. At the same time, the amount of point cloud data is large, the processing and storage costs are high, and the computational complexity is significantly increased. In addition, the point cloud itself lacks texture and color information, making it difficult to classify and identify surface damage in detail.

[0007] (3) Damage detection based on thermal imaging

[0008] Thermal imaging technology detects the thermal radiation distribution of an object and identifies areas of abnormal temperature, thereby determining the damage inside or on the surface of a structure. The technology can detect deep damage that is invisible to the naked eye, such as internal cracks, voids, or corroded areas of the material, and is particularly suitable for rapid inspection of large infrastructure in harsh environments. The main advantage of thermal imaging is that it can identify hidden damage and supplement other surface inspection methods. It is also non-contact and can complete inspections at a distance. However, its limitation is that it is highly sensitive to environmental conditions. For example, temperature changes or solar radiation may lead to false detection or missed detection. In addition, thermal imaging technology can only indirectly reflect the location of damage and lacks geometric details, making it difficult to achieve accurate quantitative analysis.

[0009] (4) Damage detection based on vibration analysis

[0010] Vibration analysis technology evaluates the health of a structure by measuring its dynamic response (such as acceleration or frequency changes). This technology usually uses modal analysis or spectral analysis to detect the location and severity of damage, and is widely used in large structures such as bridges and high-rise buildings. Vibration data can be collected by accelerometers or laser Doppler vibrometers and combined with machine learning models for automated analysis. The advantage of vibration analysis is its high sensitivity to overall structural damage and its ability to detect potential problems such as small stiffness changes. In addition, the introduction of non-contact measurement technology has improved its applicability. However, vibration analysis is difficult to accurately locate damage and needs to be combined with other technologies to provide spatial information. At the same time, its detection performance is easily affected by environmental noise, affecting the reliability of the detection results.

[0011] (5) Ultrasonic damage detection

[0012] Ultrasonic technology detects damaged areas through the propagation characteristics of high-frequency sound waves in materials. It relies on changes in wave speed, frequency or amplitude, and can efficiently analyze discontinuities within the material. It is particularly suitable for identifying cracks, voids and material degradation within concrete. The advantage of ultrasonic technology lies in its high precision, which is particularly suitable for non-destructive internal damage detection. It performs well in both laboratory and field environments. However, the method has a limited detection range and is suitable for precise analysis of small areas, but not for comprehensive monitoring of large-scale infrastructure. In addition, ultrasonic signals are easily affected by material properties and environmental conditions, and data acquisition and interpretation require high-precision equipment and expertise.

[0013] (6) Damage detection based on ground penetrating radar

[0014] Ground Penetrating Radar (GPR) is a technology based on high-frequency electromagnetic waves to detect underground or subsurface structural damage. It is widely used in the inspection of bridges, roads and building foundations. GPR analyzes the reflection characteristics of radar waves in materials to identify abnormal areas such as cracks, voids or material degradation. This technology is particularly suitable for non-destructive testing tasks and can obtain internal information without destroying the structure. The main advantage of GPR is its ability to detect hidden damage, especially when evaluating the integrity of underground and subsurface structures. In addition, its data is hierarchical and can distinguish the state of structures at different depths. However, the disadvantage of GPR is that its resolution is low and it is difficult to identify fine surface damage, such as microcracks and surface spalling. In addition, the interpretation of GPR data usually requires a combination of professional experience and complex algorithms, and is easily affected by material properties and environmental factors, such as humidity and density changes.

[0015] Although the above existing structural damage detection methods have wide applications and significant advantages, they still have some common limitations in practical use.

[0016] In summary, existing structural damage detection methods usually rely on a single type of data (such as images, point clouds or radar data), which cannot fully reflect the diversity and complexity of structural damage. For example, image processing technology lacks three-dimensional spatial information, and point clouds and radar data are difficult to capture tiny surface details or texture features, resulting in difficulty in fully reflecting damage information when dealing with complex structural damage characteristics and diverse scenarios. It is especially difficult to achieve accurate quantitative analysis of damage, such as the width, length and direction of cracks or the area and distribution of corrosion and spalling, which is crucial for accurate assessment and decision support in infrastructure health monitoring. Summary of the invention

[0017] To this end, the technical problem to be solved by the present invention is to overcome the problem that the prior art relies on a single type of data, is difficult to fully reflect damage information, and has significant limitations in three-dimensional spatial positioning and quantitative damage analysis.

[0018] In order to solve the above technical problems, the present invention provides a structural damage detection method based on multimodal fusion, comprising:

[0019] Collect 2D images and 3D point clouds of structural damage;

[0020] Use the deep learning model to segment the two-dimensional image to obtain a segmented image;

[0021] Map the 3D coordinates of the 3D point cloud to the image coordinates, establish the correspondence between the 3D point cloud and the pixels of the segmented image, and obtain the damage category label of the corresponding 3D point cloud according to the segmentation result of each pixel in the segmented image;

[0022] For the three-dimensional point cloud with the damage category label of crack, a clustering algorithm is used to cluster the three-dimensional point cloud to separate each independent crack, and then a point cloud shrinkage algorithm is used to generate the central axis of each crack, and the length, width and direction of the crack are calculated based on the central axis;

[0023] For the three-dimensional point cloud with the damage category label of planar damage, the three-dimensional point cloud is projected onto a two-dimensional plane and then the boundary polygon is extracted. The area of ​​the planar damage is calculated based on the boundary polygon. The depth of the planar damage is calculated based on the height value of the three-dimensional point cloud.

[0024] Preferably, an inspection robot equipped with a RealSense I515 radar camera is used to collect two-dimensional image data and three-dimensional point cloud data.

[0025] Preferably, the deep learning model adopts the nnU-Net model.

[0026] Preferably, when a 3D point cloud corresponds to 2D images of multiple viewing angles, a multi-angle fusion method is used to obtain the damage category label of each 3D point cloud, including:

[0027] Establish a correspondence between the three-dimensional point cloud and the segmented image pixels at each viewing angle, and obtain the probability distribution of the damage category of the three-dimensional point cloud at each viewing angle;

[0028] According to the weight of the three-dimensional point cloud at each viewing angle, the damage category distribution of the three-dimensional point cloud at each viewing angle is weighted and summed to obtain the total damage category probability distribution of the three-dimensional point cloud;

[0029] According to the total damage category probability distribution of the three-dimensional point cloud, the maximum probability method is used to obtain the damage category label of the three-dimensional point cloud.

[0030] Preferably, according to the weight of the three-dimensional point cloud at each viewing angle, the damage category distribution of the three-dimensional point cloud at each viewing angle is weighted and summed to obtain the total damage category probability distribution of the three-dimensional point cloud, and the formula is:

[0031]

[0032] in, represents the total damage category probability distribution of the 3D point cloud, J represents the total number of viewing angles of the 3D point cloud, and w j represents the weight of the j-th perspective, represents the probability distribution of damage categories at the jth viewing angle of the 3D point cloud;

[0033] The weight of the 3D point cloud at each viewing angle is determined by the angle between the normal vector of the 3D point cloud and the viewing direction. The calculation formula is:

[0034]

[0035] Among them, θ j represents the angle between the normal vector of the 3D point cloud and the direction of the jth viewing angle, and M represents the total number of viewing angles of the 3D point cloud;

[0036] According to the total damage category probability distribution of the three-dimensional point cloud, the maximum probability method is used to obtain the damage category label of the three-dimensional point cloud. The formula is:

[0037]

[0038] Among them, L(P i ) represents the category label of the i-th 3D point cloud, Represents the total damage category probability distribution of the i-th 3D point cloud.

[0039] Preferably, for the point cloud whose damage category label is crack, the DBSCAN clustering algorithm is used to cluster the three-dimensional point cloud to separate each independent crack.

[0040] Preferably, a point cloud shrinkage algorithm based on Laplace constraints is used to generate the central axis of each crack, including:

[0041] Define an energy function for each crack:

[0042]

[0043] Where E(·) represents the energy function, P' represents the set of shrunk 3D point clouds, and P i and P' i They represent the i-th 3D point cloud and the i-th shrunk 3D point cloud respectively, L represents the Laplace matrix, W L,i and W H,i Both represent the weight matrix of the i-th three-dimensional point cloud;

[0044] The set of shrunk target three-dimensional point clouds is obtained by minimizing the energy function E(P'); all shrunk target three-dimensional point clouds constitute the central axis of the current crack.

[0045] Preferably, for the three-dimensional point cloud whose damage category label is planar damage, the principal component analysis method is used to project the three-dimensional point cloud onto a two-dimensional plane, and then the α-shape algorithm is used to extract the boundary polygon of the two-dimensional point cloud.

[0046] Preferably, for the three-dimensional point cloud with the damage category label of planar damage, the distribution range of the height value of the three-dimensional point cloud is calculated, and the depth of the planar damage is calculated using the quantile method, and the formula is:

[0047] D=Q 0.95 (z)-Q 0.05 (z)

[0048] Where D represents the depth of the surface damage. z represents the height of the 3D point cloud. Q 0.95 (z) and Q 0.05 (z) represent the 95th and 5th percentiles of the height values ​​of the 3D point cloud, respectively.

[0049] The present invention also provides a structural damage detection system based on multimodal fusion, comprising:

[0050] Acquisition module, used to collect 2D images and 3D point clouds of structural damage;

[0051] A two-dimensional image segmentation module is used to segment the two-dimensional image using a deep learning model to obtain a segmented image;

[0052] A 3D point cloud label acquisition module is used to map the 3D coordinates of the 3D point cloud to the image coordinates, establish a corresponding relationship between the 3D point cloud and the segmented image pixels, and obtain the damage category label of the corresponding 3D point cloud according to the segmentation result of each pixel in the segmented image;

[0053] The crack quantification module is used to cluster the three-dimensional point cloud with the damage category label of crack using a clustering algorithm to separate each independent crack, and then use the point cloud shrinkage algorithm to generate the central axis of each crack, and calculate the length, width and direction of the crack based on the central axis;

[0054] The surface damage quantification module is used to project the three-dimensional point cloud with the damage category label of surface damage onto a two-dimensional plane, extract the boundary polygon, calculate the area of ​​the surface damage based on the boundary polygon, and calculate the depth of the surface damage based on the height value of the three-dimensional point cloud.

[0055] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0056] The structural damage detection method based on multimodal fusion described in the present invention first uses image segmentation technology to classify the damage in the two-dimensional image to obtain a segmented image, and then maps the three-dimensional point cloud to the pixels corresponding to the segmented image through a point cloud mapping algorithm. The damage category label corresponding to the three-dimensional point cloud is obtained according to the segmentation result of each pixel in the segmented image. By combining the high-resolution advantage of the two-dimensional image data with the three-dimensional spatial information of the point cloud data, all-round coverage of damage detection is achieved. In addition, based on the damage category label, the present invention extracts the structured data of cracks and planar damage according to the geometric characteristics of the damage, ensuring the accuracy and consistency of the damage detection results, and effectively improving the accuracy and robustness of damage detection. The present invention realizes the precise positioning and quantitative evaluation of damage, which not only expands the application scenarios of damage detection, but also provides new technical support for efficient and intelligent detection in the field of infrastructure health monitoring.

[0057] Furthermore, the present invention dynamically adjusts the weights, fully utilizes the two-dimensional image information under multiple perspectives, and adopts a multi-angle fusion method to obtain the damage category label of each three-dimensional point cloud to optimize the mapping result of the three-dimensional point cloud, thereby solving the problem of information missing caused by single-perspective occlusion and further improving the accuracy of damage detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0059] Figure 1 It is a flow chart of a structural damage detection method based on multi-modal fusion of the present invention;

[0060] Figure 2 It is a structural schematic diagram of a structural damage detection method based on multi-modal fusion of the present invention;

[0061] Figure 3 This is a physical picture of the inspection robot in Example 1;

[0062] Figure 4 This is an exploded view of the inspection robot in Example 1;

[0063] Figure 5 is a finite element analysis mesh diagram of the inspection robot in Example 1;

[0064] Figure 6 is a finite element analysis result diagram of the inspection robot in Example 1;

[0065] Figure 7 is a schematic diagram of crack segmentation for traditional structural damage detection, where Figure 7 (a) is the original crack image. Figure 7 (b) is the crack segmentation image;

[0066] Figure 8 is a schematic diagram of the crack segmentation result of the present invention;

[0067] Fig. 9 is a schematic diagram of the boundary extraction of planar damage of the present invention, wherein Fig. 9 (a) is the surface damage acquisition image. Fig. 9 (b) is the surface damage boundary extraction map;

[0068] Fig.10 It is a schematic diagram of the surface damage detection result of the present invention. DETAILED DESCRIPTION

[0069] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0070] Embodiment 1

[0071] Reference Figure 1 and Figure 2 As shown, the present invention provides a structural damage detection method based on multimodal fusion, comprising:

[0072] S1. Collect 2D images and 3D point clouds of structural damage.

[0073] This embodiment uses an inspection robot to collect on-site structural damage data, including but not limited to structural damage of infrastructure such as bridges, tunnels, and buildings. The collected structural damage data includes two-dimensional RGB images and three-dimensional point clouds.

[0074] The inspection robot used in this embodiment has a compact and modular structure and has excellent flexibility and adaptability. Figure 3 This is a physical picture of the inspection robot in this embodiment. Figure 4 This is an exploded view of the inspection robot in this embodiment. Figure 5 This is the finite element analysis diagram of the inspection robot in this embodiment. The inspection robot used in this embodiment is equipped with an advanced RealSense I515 radar camera, which can collect two-dimensional RGB image data and three-dimensional radar point cloud data, and realize the organic integration of visual information and three-dimensional space information. Its unique chassis design and highly flexible drive system enable it to easily cope with complex terrain, narrow space and multi-angle inspection requirements. Through precise data collection and efficient mobile performance, the coverage and data quality of defect detection are significantly improved, providing reliable data support for subsequent analysis and processing.

[0075] S2. Use a deep learning model to segment the two-dimensional image to obtain a segmented image.

[0076] Preferably, this embodiment uses the nnU-Net model as a deep learning model to segment the input two-dimensional image, identify and mark the damaged area on the surface of the structure. The model inherits the symmetric structure of U-Net, and adaptively optimizes the network's hyperparameters, data preprocessing, and training process according to the characteristics of the task.

[0077] The nnU-Net model consists of a symmetrical encoder and decoder, which extracts features through convolution operations and gradually restores the image resolution. The encoder is responsible for gradually downsampling the input image I∈R H×W×C , generate a multi-scale feature map F lThe decoder combines the features in the encoder with the upsampled features in the decoding process through skip connections to enhance the detailed expression of the segmentation results. The formula is:

[0078] F′ l =Concat(UpConv(F′ l+1 ),F l )

[0079] Among them, F l Represents the feature map output by the encoder at layer l, F' l+1 represents the feature map output by the l+1th layer decoder, F' l It represents the feature map output by the l-th layer decoder, Concat represents feature concatenation, and UpConv represents upsampling convolution.

[0080] nnU-Net automatically adjusts the network depth, feature map size, and downsampling times according to the input image size, resolution, and task requirements to generate the optimal network architecture. This adaptive mechanism avoids the tedious process of manually adjusting hyperparameters.

[0081] By analyzing the pixel spacing or resolution R of the input 2D image, the nnU-Net model normalizes the resolution to R' for uniform processing at multiple scales, as follows:

[0082]

[0083] Where R represents the resolution of the input two-dimensional image in different dimensions, for example, R = [R x ,R v ,R z ]; R' represents the standardized target resolution, which is used to ensure the consistent proportional relationship of each dimension of the image.

[0084] The network depth D' determines the number of layers of the encoder and decoder, which is selected based on the spatial size S of the input two-dimensional image, and the formula is:

[0085]

[0086] Where S = [S x ,S y ,S z ] is the size of the two-dimensional input image, that is, the number of pixels of the image; the subtracted constant, which is 2 in this embodiment, is used to avoid excessive downsampling caused by an overly deep network.

[0087] The number of channels of each layer feature map C l It is related to the number of network layers l and is automatically adjusted according to the complexity of the task. The formula is:

[0088] C l =C0 2 l

[0089] Among them, C l is the number of channels of the l-th layer network, C 0 is the initial channel number of the input two-dimensional image; l is the number of network layers, starting from 0; this multiplication mechanism can increase the feature extraction capability layer by layer.

[0090] The intensity values ​​of the input two-dimensional image are usually distributed in different ranges. In order to ensure consistency, the nnU-Net model converts the image intensity I(x, y) into a form with zero mean and unit standard deviation through a normalization method. The formula is:

[0091]

[0092] Wherein, I'(x,y) represents the normalized target image intensity, I(x,y) represents the intensity of the input two-dimensional image, and μ=E[I(x,y)] represents the mean value of the input two-dimensional image intensity; Represents the standard deviation of the input 2D image intensity.

[0093] For large-sized images, the nnU-Net model divides them into overlapping small blocks for segmentation. The size of the blocks W' and the overlap rate O are automatically calculated based on the GPU memory and image size, and the formula is:

[0094]

[0095] Among them, W max Indicates the maximum allowable size of the slice, which is determined by hardware limitations; S I Represents the full size of the input 2D image: n represents the number of tiles to be divided. Overlapping tiles are spliced ​​in the inference stage, and the edge effect is eliminated by weighted averaging to obtain a complete segmentation result.

[0096] The output layer of the nnU-Net model uses K 1×1 convolution kernels to predict the category probability distribution p(x,y)∈P for each pixel. K , generate the final segmentation result by selecting the maximum probability category

[0097]

[0098] Among them, K represents the total number of categories, p k (x,y) represents the probability distribution of the pixel point (x,y) belonging to the kth category, and S(x,y) represents the damage category of the pixel point (x,y).

[0099] The probability distribution of damage categories for all pixels constitutes a probability heat map of the two-dimensional image.

[0100] The nnU-Net model uses cross entropy loss and Dice loss for optimization, the formula is:

[0101] L=λ 1 L CE +λ 2 L Dice

[0102] Among them, L represents the total loss, L CE represents the cross entropy loss, L Dice represents the Dice loss, λ 1 and λ 2 represents the loss weight.

[0103] Cross entropy loss L CE Used to evaluate classification error, the formula is:

[0104]

[0105] Where N represents the total number of pixels, y i,k represents the true label of the kth category of the i-th pixel, P i,k represents the predicted probability of the kth category of the ith pixel.

[0106] Dice loss L Dice It is used to improve the sensitivity of the model to small damage areas. The formula is:

[0107]

[0108] Among them, y i and Represent the true label and prediction result of the i-th pixel respectively.

[0109] After model inference, the nnU-Net model generates a category probability distribution P(x,y) for each pixel. The final segmentation result S(x,y) represents the distribution of the damaged area in the two-dimensional image, which can accurately distinguish damage types such as cracks, spalling and corrosion, and provide a basis for subsequent point cloud mapping.

[0110] The present invention uses image segmentation technology to identify damaged areas from input two-dimensional images, adopts nnU-Net model as a deep learning model, and generates high-resolution damage segmentation results by multi-scale extraction and layer-by-layer decoding of image features. The model fuses the high-resolution features of the encoder during the decoding process through a jump connection mechanism, so that the output results can accurately capture the details of complex damage such as cracks, spalling and corrosion. In the training stage, the optimization strategy of joint cross entropy loss and Dice loss is adopted to improve the model's ability to recognize small damaged areas. The segmentation results are output in the form of a probability heat map, and the category of each pixel is determined by the maximum probability value. This step provides accurate damage distribution information for the subsequent point cloud mapping stage.

[0111] S3. Map the three-dimensional coordinates of the three-dimensional point cloud to the image coordinates, establish a corresponding relationship between the three-dimensional point cloud and the segmented image pixels, and obtain the damage category label of the corresponding three-dimensional point cloud according to the segmentation result of each pixel in the segmented image.

[0112] In the point cloud mapping stage, the present invention projects the segmentation result S(x,y) of the two-dimensional image into the three-dimensional point cloud space to achieve seamless transfer of damage categories from the two-dimensional image to the three-dimensional space. By combining the geometric relationship between the image and the point cloud and multi-view information, a three-dimensional segmented point cloud with damage category labels is generated.

[0113] The 3D coordinates P of each 3D point cloud i =(X i ,Y i ,Z i ) According to the camera intrinsic parameter matrix K c and the external parameter matrix [R c |t] is mapped to image coordinates (u i ,v i ), the formula is:

[0114]

[0115] Among them, K c is the camera's intrinsic parameter matrix, including the focal length f x ,f x and the principal point position (c x ,c v );R c and t are the rotation matrix and translation vector of the camera respectively; (u i ,v i ) is the three-dimensional point cloud P i The corresponding pixel coordinates in the 2D image.

[0116] According to the pixel coordinates obtained by mapping (u i ,v i), obtain the category label L(P) of the three-dimensional point cloud from the two-dimensional image segmentation result S(u,v) i ),Right now:

[0117] L(P i )=S(u i ,v i )

[0118] Through the above mapping relationship, each 3D point cloud is assigned the damage category label of its corresponding 2D image.

[0119] Preferably, in order to make full use of image information from multiple perspectives and solve the occlusion problem that may be caused by a single perspective, the present invention adopts a multi-perspective fusion method to optimize the category of point cloud points, that is, two-dimensional images of multiple perspectives are collected for a three-dimensional point cloud, and then a multi-angle fusion method is used to obtain the damage category label of each three-dimensional point cloud, including:

[0120] S3-1. Establish a corresponding relationship between the three-dimensional point cloud and the segmented image pixels at each viewing angle, and obtain the damage category probability distribution of the three-dimensional point cloud at each viewing angle.

[0121] Each 3D point cloud may be observed by 2D images from multiple perspectives. The category distribution for the jth perspective is By corresponding pixel (u j ,v j ) is obtained from the two-dimensional image probability heat map, the formula is:

[0122]

[0123] in, represents the probability distribution of damage category at the jth viewing angle of the 3D point cloud, (u i,j ,v i,j ) represents the pixels of the two-dimensional image of the j-th viewing angle of the three-dimensional point cloud.

[0124] S3-2. According to the weight of the three-dimensional point cloud at each viewing angle, the damage category distribution of the three-dimensional point cloud at each viewing angle is weighted and summed to obtain the total damage category probability distribution of the three-dimensional point cloud. The formula is:

[0125]

[0126] in, represents the total damage category probability distribution of the 3D point cloud, J represents the total number of viewing angles of the 3D point cloud, and w j represents the weight of the j-th view.

[0127] The weight of the 3D point cloud at each viewing angle is determined by the angle between the normal vector of the 3D point cloud and the viewing direction. The calculation formula is:

[0128]

[0129] Among them, w j represents the weight of the j-th perspective, θ j Represents the angle between the normal vector of the 3D point cloud and the direction of the j-th viewing angle.

[0130] S3-3. Based on the total damage category probability distribution of the three-dimensional point cloud, the maximum probability method is used to obtain the damage category label of the three-dimensional point cloud. The formula is:

[0131]

[0132] Among them, L(P i ) represents the category label of the i-th 3D point cloud, Represents the total damage category probability distribution of the i-th 3D point cloud.

[0133] After the above steps, each 3D point cloud P i are assigned corresponding category labels L(P i ), the output segmented 3D point cloud P seg The form is:

[0134] P seg ={(P i ,L(P i ))|i=1,2,......N P}

[0135] Among them, P seg Represents the set of segmented 3D point clouds, N P Indicates the total number of 3D point clouds.

[0136] In summary, after the two-dimensional image segmentation is completed, the present invention projects the segmentation result into the three-dimensional point cloud space to form a segmented point cloud. Through the camera projection model, the three-dimensional coordinates of the three-dimensional point cloud are mapped to the image plane, and the correspondence between the three-dimensional point cloud and the two-dimensional image pixels is established. The damage category information is obtained from the two-dimensional image and assigned to the three-dimensional point cloud. In order to enhance the robustness of the detection, the present invention uses multi-view images to fuse the categories of the three-dimensional point cloud, and calculates the category distribution of each point by weighted average, thereby solving the occlusion problem and information loss caused by a single perspective. In addition, the perspective weight is dynamically adjusted according to the angle between the normal vector of the point cloud point and the perspective direction, ensuring the scientific nature of the category distribution fusion. The final output segmented three-dimensional point cloud contains the spatial coordinates and category labels of each point, laying a data foundation for subsequent instance extraction.

[0137] S4. Perform damage instance level extraction according to damage categories and conduct quantitative analysis of cracks and surface damage.

[0138] In the instance-level damage extraction stage, the present invention is based on segmenting the three-dimensional point cloud P seg ,For different types of damage features, such as cracks and surface damage, we design adaptive extraction methods to parse the damage from point clouds into instantiated structural information. Through geometric analysis and clustering algorithms, we achieve accurate quantitative analysis of damage.

[0139] S4-1. For the three-dimensional point cloud with the damage category label of crack, a clustering algorithm is used to cluster the three-dimensional point cloud to separate each independent crack, and then a point cloud shrinkage algorithm is used to generate the central axis of each crack, and the length, width and direction of the crack are calculated based on the central axis.

[0140] Since the geometric characteristics of the crack are distributed linearly, this embodiment uses a point cloud shrinkage algorithm to generate the central axis of the crack, thereby extracting the main features of the crack.

[0141] S4-1-1. Use DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm to perform preliminary clustering on the crack point cloud and separate each independent crack. The formula is:

[0142] DBSCAN(P,ε,MinPts)→{C 1 ,C 2 ,......,C d}

[0143] Among them, P represents the three-dimensional point cloud, ε represents the distance threshold between points, MinPts represents the minimum point count threshold, and C d represents the dth crack point cloud cluster.

[0144] S4-1-2. Use the point cloud shrinkage algorithm to generate the central axis of each crack.

[0145] For each cluster C d , the central axis of each crack is generated using a point cloud shrinkage algorithm based on Laplace constraints, including:

[0146] Define an energy function for each crack:

[0147]

[0148] Where E(·) represents the energy function, P' represents the set of shrunk 3D point clouds, and P i and P' i They represent the i-th 3D point cloud and the i-th shrunk 3D point cloud respectively, L represents the Laplace matrix, W L,i and W H,i Both represent the weight matrix of the i-th three-dimensional point cloud;

[0149] The set of shrunk target three-dimensional point clouds is obtained by minimizing the energy function E(P'); all shrunk target three-dimensional point clouds constitute the central axis of the current crack.

[0150] S4-1-3. Calculate the length, width and direction of the crack based on the central axis.

[0151] For a crack, it corresponds to a crack point cloud cluster and generates a central axis. This step takes a crack as an example to illustrate the method of calculating the length, width and direction of the crack based on the central axis.

[0152] The length of the crack is calculated from the cumulative distance of the points on the central axis.

[0153] Assume that the central axis is N c continuous points, and the set is represented by The three-dimensional coordinates of each point are P' i =(x i ,y i ,z i ), the total length L of the crack is:

[0154]

[0155] Among them, ||P' i+1 -P' i || represents the Euclidean distance between two adjacent points on the central axis, and the formula is:

[0156]

[0157] The width of the crack is obtained by calculating the vertical distance distribution of the crack point cloud relative to its corresponding central axis.

[0158] Define the centerline point P' m and its normal vector The normal vector of the central axis point can be obtained through the PCA principal axis direction. i =(x i ,y i ,z i ), calculate its vertical distance to the central axis, the formula is:

[0159]

[0160] Among them, d i represents the minimum vertical distance from the i-th crack point cloud to the central axis, P' m represents the mth midline point, represents the normal vector of the mth midline point, Represents the projection of the normal vector of the i-th crack point cloud to the m-th central axis point.

[0161] The width W of the crack is twice the maximum value of the minimum vertical distance from the crack point cloud to the central axis, or twice the average value of the minimum vertical distance from the crack point cloud to the central axis. The formula is expressed as:

[0162]

[0163] or

[0164]

[0165] Among them, N L is the total number of crack point clouds corresponding to the current crack.

[0166] The direction of the crack is determined by the overall trend of the central axis and is obtained by fitting the principal axis directions of the central axis points.

[0167] The set of midline points is represented as Extract the first principal axis direction of the central axis point using principal component analysis The formula is:

[0168]

[0169] in, Indicates the direction of the crack, represents the centroid of the central axis, Indicates the centerline point P' i To direction Projection.

[0170] S4-1-4. Extract the structural data of the cracks.

[0171] Crack instances are represented by Instance c ={Centerline, Length, Width, Direction}, where Centerline represents the label of the crack instance, Length represents the length of the crack, Width represents the width of the crack, and Direction represents the direction of the crack.

[0172] Figure 7 is a schematic diagram of crack segmentation for traditional structural damage detection, where Figure 7 (a) is the original crack image. Figure 7 (b) in the figure is the crack segmentation image. Figure 8 It is a schematic diagram of the crack segmentation result of the present invention.

[0173] S4-2. For the three-dimensional point cloud with the damage category label of planar damage, the three-dimensional point cloud is projected onto a two-dimensional plane and then the boundary polygon is extracted. The area of ​​the planar damage is calculated based on the boundary polygon. The depth of the planar damage is calculated based on the height value of the three-dimensional point cloud.

[0174] The geometric characteristics of planar damage (such as spalling and corrosion) are relatively complex and usually exist in the form of a two-dimensional region. Therefore, this embodiment uses a method combining point cloud projection and boundary extraction to instantiate planar damage.

[0175] S4-2-1. Use principal component analysis to project the three-dimensional point cloud onto a two-dimensional plane, and then use the α-shape algorithm to extract the boundary polygons of the two-dimensional point cloud.

[0176] The principal component analysis method is used to project the surface damage point cloud onto a two-dimensional plane, which is expressed as:

[0177] P′ M = PCA(P M )

[0178] Among them, P M Represents the surface damage point cloud, P' < The two-dimensional point cloud after projection of the surface damage point cloud is represented. By retaining the first two principal components of the principal component analysis, the plane expression of the surface damage is constructed.

[0179] On a two-dimensional plane, the α-shape algorithm is used to extract the boundary polygon of the planar damage and define the α-shape:

[0180] α={(x,y)|P′ M (x,y)≤r α}

[0181] Where α represents the boundary polygon extracted by the α-shape algorithm, r α Indicates the radius of the control polygon details, P' M (x, y) represents the coordinates of the surface damage point cloud.

[0182] The planar damage region is represented by the boundary polygon generated by the α-shape algorithm, and its area can be directly calculated by geometric methods.

[0183] Assuming that the surface damage point cloud is projected onto a two-dimensional plane, the vertices of the boundary polygon are The area of ​​the bounding polygon is calculated using the shoelace formula:

[0184]

[0185] Where A represents the area of ​​the boundary polygon, that is, the area of ​​the surface damage; N B Represents the total number of vertices of the boundary polygon. In the above formula, (xN+1 ,y N+1 )=(x 1 ,y 1 ) to ensure the polygon is closed. The above formula calculates the directed area of ​​the polygon through the ordered connection of the points, and finally takes the absolute value to get the area of ​​the damaged area.

[0186] Depth is the third dimensional distribution characteristic of planar damage and is quantified by calculating the height range of the damage point cloud.

[0187] Assume that the coordinates of each point in the surface damage point cloud are (x i ,y i ,z i ) The depth is determined by the minimum value z of the surface damage point cloud in the third dimension min and the maximum value z max The difference represents:

[0188] D=z max -z min

[0189] Where D represents the depth of the surface damage, z max and z min They represent the maximum and minimum values ​​of the height of the surface damage point cloud respectively.

[0190] In order to improve robustness, this embodiment removes the influence of abnormal points by calculating the distribution range of the height value of the planar damage point cloud. The quantile method is used to calculate the depth of the planar damage, and the formula is:

[0191] D=Q 0.95 (z)-Q 0.05 (z)

[0192] Where D represents the depth of the surface damage. z represents the height value of the 3D point cloud labeled as surface damage. Q 0.95 (z) and Q 0.05 (z) represents the 95th and 5th percentiles of the height value of the 3D point cloud, which is used to eliminate the influence of extreme values.

[0193] Mean value of height distribution of surface damage point cloud and standard deviation σ z Respectively expressed as:

[0194]

[0195] Among them, N M Represents the total number of surface damage point clouds, z i Represents the height value of the i-th surface damage point cloud.

[0196] The mean and standard deviation of the height distribution of the surface damage point cloud provide statistical information of the surface damage and are used to describe the overall morphology of the surface damage.

[0197] S4-2-2. Extract structured data of surface damage.

[0198] Instance of surface damage is represented by Instance s ={BoundaryPolygon, Area, Depth}, where BoundaryPolygon represents the label of the planar damage instance, Area represents the area of ​​the planar damage, and Depth represents the depth of the planar damage.

[0199] Fig. 9 is a schematic diagram of the boundary extraction of planar damage of the present invention, wherein Fig. 9 (a) is the surface damage acquisition image. Fig. 9 (b) in the figure is the surface damage boundary extraction map. Fig.10 It is a schematic diagram of the surface damage detection result of the present invention.

[0200] The present invention designs different instance extraction methods according to the damage type to achieve accurate quantitative analysis of cracks and surface damage. For cracks, DBSCAN clustering is used to separate independent crack instances, and the central axis of the crack is generated by the Laplace constraint algorithm to extract parameters such as the length, width and direction of the crack. For surface damage, the point cloud is projected onto a two-dimensional plane through principal component analysis, and the boundary polygons are extracted in combination with the α-shape algorithm, and the area and depth of the damage are calculated. After instance-level extraction, the damage information is converted into structured data, providing a quantitative decision-making basis for infrastructure health assessment.

[0201] In summary, the structural damage detection method based on multimodal fusion described in the present invention first uses image segmentation technology to classify the damage in the two-dimensional image to obtain a segmented image, and then maps the three-dimensional point cloud to the pixels corresponding to the segmented image through a point cloud mapping algorithm. The damage category label corresponding to the three-dimensional point cloud is obtained according to the segmentation result of each pixel in the segmented image. By combining the high-resolution advantage of two-dimensional image data with the three-dimensional spatial information of point cloud data, all-round coverage of damage detection is achieved. In addition, based on the damage category label, the present invention processes cracks and surface damage separately according to the geometric characteristics of the damage, ensuring the accuracy and consistency of the damage detection results, and effectively improving the accuracy and robustness of damage detection. The present invention realizes the precise positioning and quantitative evaluation of damage, which not only expands the application scenarios of damage detection, but also provides new technical support for efficient and intelligent detection in the field of infrastructure health monitoring.

[0202] Furthermore, the present invention dynamically adjusts the weights, fully utilizes the two-dimensional image information under multiple perspectives, and adopts a multi-angle fusion method to obtain the damage category label of each three-dimensional point cloud to optimize the mapping result of the three-dimensional point cloud, thereby solving the problem of information missing caused by single-perspective occlusion and further improving the accuracy of damage detection.

[0203] The present invention is applicable to various types of structural health monitoring tasks, including but not limited to damage detection of infrastructure such as bridges, tunnels, and buildings. In addition, the core technology of this method can be expanded to other fields such as defect detection of industrial parts and three-dimensional modeling analysis, providing technical support for intelligent quality inspection and evaluation.

[0204] Embodiment 2

[0205] Based on the structural damage detection method based on multimodal fusion described in Example 1, this embodiment proposes a structural damage detection system based on multimodal fusion, including:

[0206] Acquisition module, used to collect 2D images and 3D point clouds of structural damage;

[0207] A two-dimensional image segmentation module is used to segment the two-dimensional image using a deep learning model to obtain a segmented image;

[0208] A 3D point cloud label acquisition module is used to map the 3D coordinates of the 3D point cloud to the image coordinates, establish a corresponding relationship between the 3D point cloud and the segmented image pixels, and obtain the damage category label of the corresponding 3D point cloud according to the segmentation result of each pixel in the segmented image;

[0209] The crack quantification module is used to cluster the three-dimensional point cloud with the damage category label of crack using a clustering algorithm to separate each independent crack, and then use the point cloud shrinkage algorithm to generate the central axis of each crack, and calculate the length, width and direction of the crack based on the central axis;

[0210] The surface damage quantification module is used to project the three-dimensional point cloud with the damage category label of surface damage onto a two-dimensional plane, extract the boundary polygon, calculate the area of ​​the surface damage based on the boundary polygon, and calculate the depth of the surface damage based on the height value of the three-dimensional point cloud.

[0211] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0212] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0213] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0214] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0215] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A structural damage detection method based on multimodal fusion, characterized in that: include: Collect 2D images and 3D point clouds of structural damage; Use the deep learning model to segment the two-dimensional image to obtain a segmented image; Map the 3D coordinates of the 3D point cloud to the image coordinates, establish the correspondence between the 3D point cloud and the pixels of the segmented image, and obtain the damage category label of the corresponding 3D point cloud according to the segmentation result of each pixel in the segmented image; For the three-dimensional point cloud with the damage category label of crack, a clustering algorithm is used to cluster the three-dimensional point cloud to separate each independent crack, and then a point cloud shrinkage algorithm is used to generate the central axis of each crack, and the length, width and direction of the crack are calculated based on the central axis; For the three-dimensional point cloud with the damage category label of planar damage, the three-dimensional point cloud is projected onto a two-dimensional plane and then the boundary polygon is extracted. The area of ​​the planar damage is calculated based on the boundary polygon. The depth of the planar damage is calculated based on the height value of the three-dimensional point cloud.

2. The structural damage detection method based on multimodal fusion according to claim 1 is characterized in that: An inspection robot equipped with a RealSense I515 radar camera is used to collect two-dimensional image data and three-dimensional point cloud data.

3. The structural damage detection method based on multimodal fusion according to claim 1 is characterized in that: The deep learning model adopts the nnU-Net model.

4. The structural damage detection method based on multimodal fusion according to claim 1 is characterized in that: When a 3D point cloud corresponds to 2D images from multiple perspectives, a multi-angle fusion method is used to obtain the damage category label of each 3D point cloud, including: Establish a correspondence between the three-dimensional point cloud and the segmented image pixels at each viewing angle, and obtain the probability distribution of the damage category of the three-dimensional point cloud at each viewing angle; According to the weight of the three-dimensional point cloud at each viewing angle, the damage category distribution of the three-dimensional point cloud at each viewing angle is weighted and summed to obtain the total damage category probability distribution of the three-dimensional point cloud; According to the total damage category probability distribution of the three-dimensional point cloud, the maximum probability method is used to obtain the damage category label of the three-dimensional point cloud.

5. The method for structural damage detection based on multimodal fusion according to claim 4, characterized in that: According to the weight of the 3D point cloud at each viewing angle, the damage category distribution of the 3D point cloud at each viewing angle is weighted and summed to obtain the total damage category probability distribution of the 3D point cloud. The formula is: in, represents the total damage category probability distribution of the 3D point cloud, J represents the total number of viewing angles of the 3D point cloud, and w j represents the weight of the j-th perspective, represents the probability distribution of damage categories at the jth viewing angle of the 3D point cloud; The weight of the 3D point cloud at each viewing angle is determined by the angle between the normal vector of the 3D point cloud and the viewing direction. The calculation formula is: Among them, θ j represents the angle between the normal vector of the 3D point cloud and the direction of the jth viewing angle, and M represents the total number of viewing angles of the 3D point cloud; According to the total damage category probability distribution of the three-dimensional point cloud, the maximum probability method is used to obtain the damage category label of the three-dimensional point cloud. The formula is: Among them, L(P i ) represents the category label of the i-th 3D point cloud, Represents the total damage category probability distribution of the i-th 3D point cloud.

6. The structural damage detection method based on multimodal fusion according to claim 1 is characterized in that: For point clouds with damage category labels as cracks, the DBSCAN clustering algorithm is used to cluster the three-dimensional point clouds and separate each independent crack.

7. The structural damage detection method based on multimodal fusion according to claim 1 is characterized in that: The central axis of each crack is generated using a point cloud shrinkage algorithm based on Laplace constraints, including: Define an energy function for each crack: Where E(·) represents the energy function, P' represents the set of shrunk 3D point clouds, and P i and P' i They represent the i-th 3D point cloud and the i-th shrunk 3D point cloud respectively, L represents the Laplace matrix, W L,i and W H,i Both represent the weight matrix of the i-th three-dimensional point cloud; The set of shrunk target three-dimensional point clouds is obtained by minimizing the energy function E(P'); all shrunk target three-dimensional point clouds constitute the central axis of the current crack.

8. The structural damage detection method based on multimodal fusion according to claim 1 is characterized in that: For the three-dimensional point cloud with the damage category label of planar damage, the principal component analysis method is used to project the three-dimensional point cloud onto a two-dimensional plane, and then the α-shape algorithm is used to extract the boundary polygon of the two-dimensional point cloud.

9. The method for structural damage detection based on multimodal fusion according to claim 1, characterized in that: For the three-dimensional point cloud with the damage category label of planar damage, the distribution range of the height value of the three-dimensional point cloud is calculated, and the depth of the planar damage is calculated using the quantile method. The formula is: D=Q 0.95 (z)-Q 0.05 (z) Where D represents the depth of the surface damage. z represents the height of the three-dimensional point cloud. Q 0.95 (z) and Q 0.05 (z) represent the 95th and 5th percentiles of the height values ​​of the 3D point cloud, respectively.

10. A structural damage detection system based on multimodal fusion, characterized in that: include: Acquisition module, used to collect 2D images and 3D point clouds of structural damage; A two-dimensional image segmentation module is used to segment the two-dimensional image using a deep learning model to obtain a segmented image; A 3D point cloud label acquisition module is used to map the 3D coordinates of the 3D point cloud to the image coordinates, establish a corresponding relationship between the 3D point cloud and the segmented image pixels, and obtain the damage category label of the corresponding 3D point cloud according to the segmentation result of each pixel in the segmented image; The crack quantification module is used to cluster the three-dimensional point cloud with the damage category label of crack using a clustering algorithm to separate each independent crack, and then use the point cloud shrinkage algorithm to generate the central axis of each crack, and calculate the length, width and direction of the crack based on the central axis; The surface damage quantification module is used to project the three-dimensional point cloud with the damage category label of surface damage onto a two-dimensional plane, extract the boundary polygon, calculate the area of ​​the surface damage based on the boundary polygon, and calculate the depth of the surface damage based on the height value of the three-dimensional point cloud.

Images