Building structure damage detection method, system, equipment and medium thereof

Abstract

The invention relates to the technical field of building engineering, in particular to a building structure damage detection method, system and equipment and a medium thereof. The method comprises the following steps: collecting multi-modal data of a building structure, and preprocessing the multi-modal data; constructing a three-dimensional model of the building; performing damage identification and classification on the multi-modal data through a damage detection classification model to obtain damage information; mapping the damage information to a three-dimensional model, and carrying out visual monitoring on the building damage; analyzing the multi-modal data through the damage prediction model, and performing damage prediction on the building structure to obtain a prediction result; and generating a damage assessment report according to the damage information and the prediction result. The building structure damage detection method provided by the invention has the advantages of high detection efficiency, high detection precision, real-time early warning and high safety.

Description

Technical Field

[0001] The present invention relates to the technical field of construction engineering, and particularly relates to a method, system, device and medium for detecting structural damage of buildings. Background Art

[0002] With the acceleration of the urbanization process and the extensive construction of infrastructure such as high-rise buildings, bridges, and tunnels, the safety issue of buildings has become increasingly important. During its service life, a building is often affected by factors such as the external environment, natural disasters, and long-term loads, and gradually shows varying degrees of damage, such as cracks, spalling, and steel bar corrosion. If these damages are not detected and repaired in time, it may lead to a decline in structural performance and even cause serious safety accidents. Therefore, regular monitoring and assessment of building damage are the key means to ensure its safety and extend its service life.

[0003] In traditional building safety monitoring, manual inspection is the most common method. Usually, professional technicians use means such as visual inspection and physical measurement instruments to check buildings. However, manual inspection has many limitations. Manual inspection is often time-consuming and laborious. Especially for large building complexes, bridges or high-rise buildings, it is difficult to achieve comprehensive and frequent inspections. Limited by the manual judgment ability, the results of visual inspection have subjective errors. Especially for the identification of micro-cracks or early damages, it is difficult, which may lead to damages not being detected in time. In high-altitude operations or complex environments, manual detection also has great safety risks. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0005] Therefore, the present invention provides a method for detecting structural damage of buildings, which has fast detection efficiency, high detection accuracy, can give real-time warnings, and has high safety.

[0006] According to the method for detecting structural damage of buildings according to the embodiments of the present invention, the method includes the following steps:

[0007] S1, collecting multi-modal data of the building structure and preprocessing the multi-modal data;

[0008] S2, constructing a three-dimensional model of the building;

[0009] S3, performing damage identification and classification on the multi-modal data through a damage detection and classification model to obtain damage information;

[0010] S4, mapping the damage information on the three-dimensional model to visually monitor the building damage;

[0011] S5. Analyze multimodal data through a damage prediction model to predict the damage of the building structure and obtain a prediction result.

[0012] S6. Generate a damage assessment report based on the damage information and the prediction result.

[0013] The beneficial effects of the present invention are as follows:

[0014] 1. For the building structure damage detection method of the present invention, a three-dimensional model of the building structure is generated by lidar, and damage information is mapped on the three-dimensional model, providing an intuitive three-dimensional visualization display. The three-dimensional model makes the spatial distribution, depth, and location of the damage clearer, helping managers quickly understand the damage situation. Especially when facing complex structures, three-dimensional visualization provides more decision-making basis than planar images.

[0015] 2. The present invention continuously monitors the damage evolution of the building and has a dynamic trend analysis function. By analyzing multimodal data through a damage prediction model, the expansion trend of the damage is predicted, which can help managers identify potential risks in advance and prevent the further deterioration of building safety problems.

[0016] 3. The building structure damage detection method of the present invention can not only identify damage but also automatically measure the size information and location of the damage. With the help of point cloud data, measurements can be accurate to the millimeter level, which is especially suitable for the measurement of fine cracks and the evaluation of complex structures.

[0017] According to an embodiment of the present invention, in the step S1, an RGB image, a thermal distribution image, and point cloud data on the surface of the building structure are respectively obtained by using an image acquisition unit, an infrared thermal imager, and a lidar, and the RGB image, the thermal distribution image, and the point cloud data form the multimodal data.

[0018] The preprocessing of the multimodal data includes:

[0019] Denoise the RGB image and the thermal distribution image.

[0020] Perform downsampling processing on the point cloud data.

[0021] According to an embodiment of the present invention, the step S3 includes the following steps:

[0022] S31. Process the denoised RGB image through an edge detection algorithm to extract damage edge features.

[0023] S32. Process the damage edge features to obtain the width of the damage.

[0024] S33. Perform fusion processing on the RGB image, the infrared thermal image, and the point cloud data to obtain multi-dimensional spatial point cloud data.

[0025] S34. Obtain the point cloud data of the damage edge features, and use the surface fitting technology to obtain the depth of the damage;

[0026] S35. Through the PointNet neural network, combined with the building structure, segment the multi-dimensional spatial point cloud data to obtain the point cloud data sets of each structural component respectively;

[0027] S36. Perform clustering analysis on the texture features and block features in the point cloud data of each structural component to obtain the damage positions and sizes of each structural component;

[0028] S37. According to the damage positions and sizes of each structural component, based on the normal direction and depth information of the point cloud, use the voxel stretching method to extend along the depth direction, expand along the horizontal direction, and merge the voxels to form a spatial structure body of the damage;

[0029] S38. Classify the damage degree according to the width and depth of the damage to obtain the classification result;

[0030] Wherein, the damage information includes damage category, damage position, damage size and damage degree.

[0031] According to an embodiment of the present invention, the step S5 specifically includes the following steps:

[0032] Analyze the historical multi-modal data of the building structure to identify the evolution trend of the damage over time;

[0033] Analyze the evolution trend of the damage over time through the damage prediction model, perform damage prediction on the building structure, and obtain the prediction result. The prediction formula is: h t = O t *tanh(C t );

[0034] Wherein, h t is the output of the damage prediction model, used to predict the damage state at a future moment, O t is the output gate, and C t is the memory cell state;

[0035] Perform risk assessment on the prediction result to determine whether to trigger a warning signal.

[0036] According to an embodiment of the present invention, the step S33 specifically includes the following steps:

[0037] S331. Perform two-dimensional image fusion on the RGB image and the infrared thermal image to obtain a fusion image containing RGB color and temperature information;

[0038] S332, use the LBP local binary pattern feature extraction operator to extract the texture features of the fused image;

[0039] S333, use the sobel operator to extract the contour features of the fused image and convert the contour features into binary features;

[0040] S335, fuse based on the texture features and binary features to form block features;

[0041] S336, based on the temperature information of the fused image, perform differencing using the background temperature to obtain the temperature features of the image;

[0042] S337, combine the relationship between the temperature features of the image and the damage depth to perform damage depth feature conversion;

[0043] S338, synthesize the converted damage depth features with the fused image to form a multi-dimensional fused image;

[0044] S339, align and register the multi-dimensional fused image with the point cloud data, map the multi-dimensional information onto the three-dimensional point cloud, and obtain the multi-dimensional space point cloud data.

[0045] According to an embodiment of the present invention, the step S341 specifically includes the following steps:

[0046] Based on the resolution of the RGB image, perform super-resolution reconstruction on the infrared thermal image using bilinear interpolation, and the resolution of the RGB image is equal to the resolution of the infrared thermal image after reconstruction;

[0047] Unify the coordinate systems of the RGB image and the infrared thermal image after reconstruction, and use the multi-scale fusion image fusion algorithm to process the coordinate systems of the RGB image and the infrared thermal image after reconstruction to obtain a fused image containing RGB color and temperature information.

[0048] According to an embodiment of the present invention, the method also supports real-time interaction operations between the user and the three-dimensional model and the display of dynamic information.

[0049] According to a building structure damage detection system of an embodiment of the present invention, the detection system includes:

[0050] A data acquisition and processing module, which acquires multi-modal data of the building structure and preprocesses the multi-modal data;

[0051] A construction module, which constructs a three-dimensional model of the building;

[0052] A damage detection and classification module, which performs damage identification and classification on the multi-modal data through a damage detection and classification model to obtain damage information;

[0053] A visualization display module that maps the damage information onto the 3D model for visual monitoring of building damage;

[0054] A prediction module that analyzes multi-modal data through a damage prediction model to predict the damage of the building structure and obtain a prediction result;

[0055] An evaluation report generation module that generates a damage evaluation report based on the damage information and the prediction result.

[0056] A computer device according to an embodiment of the present invention includes:

[0057] A processor;

[0058] A memory for storing executable instructions;

[0059] Wherein, the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the building structure damage detection method as described above.

[0060] A computer-readable storage medium according to an embodiment of the present invention, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to implement the building structure damage detection method as described above.

[0061] Other features and advantages of the present invention will be described in the following specification, and in part will be obvious from the specification, or will be understood by implementing the present invention. The objectives and other advantages of the present invention are realized and obtained by the structures specifically pointed out in the specification, the claims, and the drawings.

[0062] To make the above objectives, features, and advantages of the present invention more obvious and understandable, the following specific preferred embodiments are given, and in conjunction with the accompanying drawings, the detailed description is as follows. Description of the Drawings

[0063] The present invention will be further described below in conjunction with the drawings and embodiments.

[0064] Figure 1 It is a schematic flowchart of the method according to Embodiment 1 of the present invention.

[0065] Figure 2 It is a schematic structural diagram of the detection system according to Embodiment 2 of the present invention.

[0066] Figure 3 It is a schematic structural diagram of the data acquisition and processing module according to Embodiment 2 of the present invention.

[0067] Figure 4 It is a schematic structural diagram of the damage detection and classification module according to Embodiment 2 of the present invention.

[0068] Figure 5It is a schematic structural diagram of the visualization display module according to the second embodiment of the present invention.

[0069] Figure 6 It is a schematic structural diagram of the prediction module according to the second embodiment of the present invention.

[0070] Figure 7 It is a schematic structural diagram of the evaluation report generation module according to the second embodiment of the present invention.

[0071] Figure 8 It is a schematic structural diagram of the dynamic interaction module according to the second embodiment of the present invention.

[0072] Figure 9 It is a schematic structural diagram of the computer device according to the third embodiment of the present invention.

[0073] In the figure, 10 is the computer device; 1002 is the processor; 1004 is the memory; 1006 is the transmission device. Detailed implementation manners

[0074] Now, the present invention will be further described in detail with reference to the accompanying drawings. These drawings are all simplified schematic diagrams, only illustrating the basic structure of the present invention in a schematic manner, so they only show the components related to the present invention.

[0075] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the present invention. In addition, the features defined as "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, the meaning of "a plurality" is two or more.

[0076] In the description of the present invention, it should be noted that, unless otherwise clearly defined and limited, the terms "mounted", "connected" and "coupled" should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the internal communication of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0077] Embodiment 1

[0078] An embodiment of the present application provides a method for detecting building structure damage, as Figure 1 shown, the method includes the following steps:

[0079] S1, collect multimodal data of the building structure and preprocess the multimodal data; wherein, the multimodal data includes: RGB images, thermal distribution images, and point cloud data.

[0080] Specifically, an image acquisition unit is used to obtain RGB images of the building structure surface for detecting damage on the building structure surface; an infrared thermal imager is used to obtain thermal distribution images of the building structure surface. Structural damage inside the building, such as cracks or material fatigue, often causes local temperature anomalies, and the infrared imager can identify thermal anomalies caused by internal damage to the building structure; a lidar is used to obtain point cloud data of the building structure surface. By scanning the building structure surface with a lidar, high-precision point cloud data is generated.

[0081] Preprocessing the multimodal data includes:

[0082] Denoise the RGB images and thermal distribution images, and use the non-local means filtering algorithm to reduce image noise while retaining the edges and details of the images. The non-local means filtering algorithm calculates weights based on the similarity in the pixel neighborhood, thereby effectively removing high-frequency noise. The specific formula is as follows;

[0083]

[0084] where I(x,y) is the original value of the pixel point (x,y), h is the smoothing parameter, and N(x,y) is the pixel neighborhood; I denoised (x,y) is the value of the pixel point (x,y) after denoising, that is, the smoothed pixel value obtained after formula calculation; I(p) is the original value of the neighborhood pixel point p; Z(x,y) is the normalization factor used to normalize the sum of weights to ensure the rationality of the calculation result.

[0085] Perform downsampling processing on the point cloud data to improve the calculation efficiency of the point cloud data. The point cloud data is used to capture the global structure and surface details of the building and provide key data input for 3D modeling. The downsampling optimization formula is:

[0086] P sampled ={p i |p i ∈P, p i is the point closest to the voxel center}

[0087] where P is the original point cloud data, and P sampled is the downsampled point cloud.

[0088] S2. Construct a three-dimensional model of the building; further, generate a three-dimensional model of the building through the Poisson surface reconstruction algorithm using the downsampled point cloud data, and the reconstructed surface can truly display the geometric shape and details of the building structure. The basic form of the Poisson equation is;

[0089]

[0090] where, represents the Laplace operator in the Poisson equation, which represents the divergence of the gradient of the scalar field φ and is used to describe the surface morphology of the three-dimensional model; is the summation symbol, indicating the accumulation of all terms from i = 1 to n in the point cloud data, and n i is the weight coefficient, representing the contribution size or importance of the i-th point in the point cloud data, and this value is usually related to the density of the point or specific physical properties; δ(x - x i ) is the Dirac δ function, representing the impulse generated at the position of point x i and is used to map the positions of discrete points in the point cloud to a continuous spatial distribution as a constraint condition for constructing the implicit function ; x is the current calculation position, representing the position of any point in three-dimensional space; x i is the specific coordinate of the i-th discrete point in the point cloud.

[0091] S3. Use the damage detection and classification model to identify and classify damage in multi-modal data to obtain damage information; specifically including:

[0092] S31. Process the denoised RGB image through the edge detection algorithm to extract damage edge features; further, first calculate the image gradient through the Sobel operator, and the calculation formula is;

[0093]

[0094] where, G x and G y are the gradients of the image in the horizontal and vertical directions respectively;

[0095] Then extract the damage edge features through non-maximum suppression and double-threshold method.

[0096] S32. Process the damage edge features to obtain the width of the damage, and the calculation formula is:

[0097]

[0098] where, p1 and p2 are the coordinates of the crack edge points respectively;

[0099] S33. Perform fusion processing on the RGB image, infrared thermal image, and point cloud data to obtain multi-dimensional spatial point cloud data. The obtained multi-dimensional spatial point cloud data can more comprehensively reflect the overall health status of the building and improve the effectiveness of monitoring.

[0100] S34. Obtain the point cloud data of the damage edge features, and use the surface fitting technology to obtain the depth of the damage.

[0101] S35. Through the PointNet neural network, segment the multi-dimensional spatial point cloud data in combination with the building structure to obtain the point cloud data sets of each structural component respectively.

[0102] S36. Perform clustering analysis on the texture features and block features in the point cloud data of each structural component to obtain the damage location and size of each structural component.

[0103] S37. According to the damage location and size of each structural component, based on the normal direction and depth information of the point cloud, use the method of voxel stretching to extend along the depth direction, expand along the horizontal direction, and merge the voxels (abbreviation of Volume Pixel, which is the smallest unit of digital data segmentation in three-dimensional space) to form a spatial structure body of the damage.

[0104] S38. Classify the damage degree according to the width and depth of the damage to obtain the classification result.

[0105] The classification results include:

[0106] Deep damage: When the depth > a (recommended value is 5), it is classified as deep damage.

[0107] Surface damage: When the width < b (recommended value is 0.5) and the depth < c (recommended value is 2), it is classified as surface damage.

[0108] General damage: Other damages where both the width and depth are between the above thresholds are classified as general damage. Among them, the damage information includes damage category, damage location, damage size, and damage degree, and the damage categories include cracks, spalling, corrosion, and deformation.

[0109] It should be noted that by fusing RGB images, infrared thermal images, and point cloud data, and combining visible light textures, thermal distributions, and three-dimensional geometric information, the structural state of the building can be reflected in multiple dimensions, enhancing the comprehensiveness and accuracy of monitoring. Using the PointNet neural network to automatically segment multi-dimensional point cloud data can efficiently disassemble the building into individual components (such as beams, columns, etc.), and accurately identify independent point clouds in combination with the structural characteristics of the building, providing a reliable basis for subsequent feature extraction and damage analysis, especially performing well in complex building environments. Conducting clustering analysis on the texture and block features in the component point cloud data can accurately locate the damage position and size of each component, and at the same time quantify the damage, providing data support for subsequent modeling.

[0110] S4. Map the damage information onto the three-dimensional model to visually monitor the building damage;

[0111] The damage category, damage position, damage size, and damage degree are mapped on the three-dimensional model. The damage information is mapped onto the three-dimensional model through color coding and is based on the following normalization formula for color mapping;

[0112]

[0113] d i is the original damage value, that is, the specific damage measurement value before normalization

[0114] where s i is the damage value after normalization, d min and d max are the minimum and maximum damage values respectively;

[0115] During visual monitoring, the damage evolution process of the building at different time points can be displayed. Users can select different time points through the time axis to view the damage changes and dynamically track the expansion trend of the damage; in addition, through the thermal distribution map, a dynamic thermal map can be generated according to the damage severity, intuitively showing the damage risks in each area of the building. The color of the dynamic thermal map is encoded according to the damage value, and the mapping formula is;

[0116] C(s i ) = C min +(C max -C min )·s i

[0117] where; C(S i ) is the color value mapped from the damage value S i after normalization to the dynamic thermal map, representing the color intensity of the damage area in the thermal map; C min is the minimum color value, corresponding to the lower limit of the color intensity in the thermal map; C maxThe maximum color value, corresponding to the upper limit of the color intensity in the heat map; S i The normalized damage value, usually ranging from [0, 1], representing the relative damage degree.

[0118] Through the dynamic heat map, users can quickly identify the areas where damage is concentrated, facilitating timely maintenance decisions.

[0119] S5. Analyze multi-modal data through the damage prediction model to predict the damage of the building structure and obtain the prediction result; specifically, it includes the following steps:

[0120] Analyze the historical multi-modal data of the building structure to identify the evolution trend of damage over time;

[0121] Analyze the evolution trend of damage over time through the damage prediction model to predict the damage of the building structure and obtain the prediction result. The prediction formula is: h t = O t * tanh(C t ) ;

[0122] Among them, h t is the output of the damage prediction model, used to predict the damage state at future moments, O t is the output gate, C t is the memory cell state, storing historical damage information and long-term dependency relationships;

[0123] Conduct a risk assessment on the prediction result to determine whether to trigger a warning signal.

[0124] S6. Generate a damage assessment report based on the damage information and prediction result. The damage assessment report includes a visual display of the damage trend. Users can intuitively view the damage expansion rate, trend changes of the building, and predicted high-risk areas. Further, according to the preset rule library and combined with the safety standards of the building structure, a repair plan with a higher priority will be generated. By analyzing the type, location, severity of the damage and the prediction result, the rule engine will automatically recommend repair measures. Each type of damage corresponds to different repair suggestions and is displayed in the report.

[0125] In the embodiment, step S33 specifically includes the following steps:

[0126] S331. Perform two-dimensional image fusion on the RGB image and the infrared thermal image to obtain a fusion image containing RGB color and temperature information; specifically, it includes the following steps:

[0127] Based on the resolution of the RGB image, perform super-resolution reconstruction on the infrared thermal image using bilinear interpolation. The resolution of the RGB image is equal to the resolution of the reconstructed infrared thermal image;

[0128] Unify the coordinate systems of the RGB image and the reconstructed infrared thermal image, and use a multi-scale fusion image fusion algorithm to process the coordinate systems of the RGB image and the reconstructed infrared thermal image to obtain a fusion image containing RGB color and temperature information.

[0129] It should be noted that the coordinate systems of the RGB image and the reconstructed infrared thermal image are unified through manual registration.

[0130] S332, Use the LBP local binary feature extraction operator to extract the texture features of the fusion image;

[0131] S333, Use the sobel operator to extract the contour features of the fusion image and convert the contour features into binary features;

[0132] S335, Based on the texture features and binary features for fusion to form block features;

[0133] S336, Based on the temperature information of the fusion image, use the background temperature for differentiation to obtain the temperature features of the image;

[0134] S337, Combine the relationship between the temperature features of the image and the damage depth to perform damage depth feature conversion;

[0135] S338, Synthesize the converted damage depth features with the fusion image to form a multi-dimensional fusion image;

[0136] S339, Align and register the multi-dimensional fusion image with the point cloud data, map the multi-dimensional information to the 3D point cloud, and obtain the multi-dimensional space point cloud data. The multi-dimensional space point cloud data includes multi-dimensional information such as (x, y, z) space coordinate information, color, temperature, texture, contour, block, depth, etc.

[0137] In step S34, the acquisition of the point cloud data of the damage edge features includes:

[0138] Project the point cloud data onto the RGB image to establish the correspondence between the point cloud and the image pixels;

[0139] Match the damage edge obtained by edge detection with the projected point cloud. Further, the matching method searches for the point cloud closest to the damage edge based on the distance threshold and uses it as the point cloud data of the damage edge features.

[0140] In an embodiment, the method also supports real-time interaction operations between the user and the 3D model and the display of dynamic information. For example, the user can control the perspective transformation of the 3D model through the interface, and operations such as rotation, scaling, and translation can be achieved. This function is realized through a perspective transformation matrix. To control the scaling of the model, that is, to control the enlargement or reduction of the model, and through the user's dragging operation, the translation of the model can be achieved, that is, to control the movement of the model on the plane. All the user's interaction operations will generate a new perspective transformation matrix through matrix combination and act on the rendering process of the 3D model in real time. The user can also select different time nodes through the timeline to view the damage states of the building at different time periods and compare the changes in damage. At the same time, based on the prediction results of the time series analysis module, the user can view the possible development trends of future damage. On the 3D model, the system dynamically displays the severity of the damage through a heat map, and the color mapping is associated with the degree of damage. When the user performs interaction operations, the user can observe the update of the heat map in real time as it changes with time or perspective.

[0141] In summary, for the building structure damage detection method of the present invention, a 3D model of the building structure is generated by lidar, and damage information is mapped on the 3D model, providing an intuitive 3D visualization display. The 3D model makes the spatial distribution, depth, and location of the damage clearer, helping managers quickly understand the damage situation. Especially when facing complex structures, 3D visualization provides more decision-making basis than planar images. The present invention continuously monitors the damage evolution of the building and has a dynamic trend analysis function. By analyzing multi-modal data through a damage prediction model, the expansion trend of the damage is predicted, which can help managers identify potential risks in advance and prevent the further deterioration of building safety problems. The building structure damage detection method of the present invention can not only identify damage but also automatically measure the size information and location of the damage. With the help of point cloud data, the measurement can be accurate to the millimeter level, which is especially suitable for the measurement of fine cracks and the evaluation of complex structures.

[0142] Embodiment 2

[0143] Based on the same inventive concept as a building structure damage detection method in the foregoing embodiment, an embodiment of the present application provides a building structure damage detection system, as Figure 2 shown, the detection system includes:

[0144] A data acquisition and processing module, which acquires multi-modal data of the building structure and preprocesses the multi-modal data;

[0145] A construction module, which constructs a 3D model of the building. Further, the 3D model of the building is generated by the Poisson surface reconstruction algorithm through the downsampled point cloud data, and the reconstructed surface can truly display the geometric shape and details of the building structure. The basic form of the Poisson equation is;

[0146]

[0147] Among them, represents the Laplace operator in the Poisson equation, representing the divergence of the gradient of the scalar field φ. It is used to describe the surface morphology of a three-dimensional model; is the summation symbol, indicating the accumulation of all terms from i = 1 to n in the point cloud data, where n i is the weight coefficient, representing the contribution size or importance of the i-th point in the point cloud data. This value is usually related to the density of the points or specific physical properties; δ(x - x i ) Dirac delta function, representing the impulse generated at the position of point x i to map the positions of discrete points in the point cloud to a continuous spatial distribution, serving as a constraint condition for constructing the implicit function ; x is the current calculation position, representing the position of any point in three-dimensional space; x i is the specific coordinate of the i-th discrete point in the point cloud.

[0148] Damage detection and classification module, which performs damage identification and classification on multi-modal data through a damage detection and classification model to obtain damage information;

[0149] Visualization display module, which maps the damage information onto the three-dimensional model to visually monitor the building damage;

[0150] Prediction module, which analyzes multi-modal data through a damage prediction model to predict the damage of the building structure and obtain a prediction result;

[0151] Evaluation report generation module, which generates a damage evaluation report based on the damage information and prediction result.

[0152] In the embodiment, as Figure 3 shown, the data acquisition and processing module includes:

[0153] Image acquisition unit, which acquires the RGB image of the building structure surface for detecting the damage on the building structure surface.

[0154] Infrared thermal imager, which acquires the thermal distribution image of the building structure surface. Structural damages inside the building such as cracks or material fatigue often cause local temperature anomalies, and the infrared imager can identify the thermal anomalies caused by internal damages of the building structure.

[0155] LiDAR, which acquires the point cloud data of the building structure surface. By scanning the building structure surface with LiDAR, high-precision point cloud data is generated.

[0156] Control unit, the image acquisition unit, the infrared thermal imager, and the LiDAR are all connected to the control unit, and the control unit is used to control the working states of the image acquisition unit, the infrared thermal imager, and the LiDAR.

[0157] A data processing unit for preprocessing multimodal data, and the preprocessing includes:

[0158] Denoising the RGB image and the thermal distribution image, and using the non-local means filtering algorithm to reduce image noise while retaining the edges and details of the image. The non-local means filtering algorithm calculates weights based on the similarity in the pixel neighborhood, thereby effectively removing high-frequency noise. The specific formula is as follows;

[0159]

[0160] where I(x, y) is the original value of the pixel point (x, y), h is the smoothing parameter, and N(x, y) is the pixel neighborhood; I denoised (x, y) is the value of the pixel point (x, y) after denoising, that is, the smoothed pixel value obtained after formula calculation; I(p) is the original value of the neighborhood pixel point p; Z(x, y) is the normalization factor for normalizing the sum of weights to ensure the rationality of the calculation result.

[0161] Performing downsampling on the point cloud data to improve the calculation efficiency of the point cloud data. The point cloud data is used to capture the global structure and surface details of the building and provide key data input for 3D modeling. The downsampling optimization formula is:

[0162] P sampled ={p i |p i ∈P, and pi is the point closest to the voxel center}

[0163] where P is the original point cloud data and P sampled is the downsampled point cloud.

[0164] It should be noted that the image acquisition unit and the infrared thermal imager are integrated together, and the control unit will control the image acquisition unit and the infrared thermal imager to collect synchronously to facilitate the subsequent fusion processing of multimodal data.

[0165] In the embodiment, as Figure 4 shown, the damage detection and classification module includes:

[0166] An edge damage extraction unit that processes the denoised RGB image through an edge detection algorithm to extract damage edge features; further, first calculates the image gradient through the Sobel operator, and the calculation formula is;

[0167]

[0168] where G x and G y are the gradients of the image in the horizontal and vertical directions respectively;

[0169] Then, the damage edge features are extracted by non-maximum suppression and double-threshold method.

[0170] A damage size calculation unit processes the damage edge features to obtain the width of the damage. The calculation formula is:

[0171]

[0172] where p1 and p2 are the coordinates of the crack edge points respectively;

[0173] The point cloud data of the damage edge features is obtained, and the depth of the damage is obtained by using the surface fitting technology.

[0174] A multi-modal data fusion unit fuses the RGB image, infrared thermal image and point cloud data to obtain multi-dimensional spatial point cloud data. Obtaining multi-dimensional spatial point cloud data can more comprehensively reflect the overall health status of the building and improve the effectiveness of monitoring.

[0175] A point cloud segmentation unit segments the multi-dimensional spatial point cloud data through the PointNet neural network in combination with the building structure to obtain the point cloud data sets of each structural component respectively.

[0176] A point cloud clustering analysis unit performs clustering analysis on the texture features and block features in the point cloud data of each structural component to obtain the damage positions and sizes of each structural component.

[0177] A spatial structure construction unit extends along the depth direction in a voxel stretching manner based on the damage positions and sizes of each structural component, expands along the horizontal direction, and merges the voxels to form a spatial structure of the damage according to the normal direction and depth information of the point cloud.

[0178] A classification unit classifies the damage degree according to the width and depth of the damage to obtain the classification result. The classification results include:

[0179] Deep damage: When the depth > a (recommended value is 5), it is classified as deep damage.

[0180] Surface damage: When the width < b (recommended value is 0.5) and the depth < c (recommended value is 2), it is classified as surface damage.

[0181] General damage: Other damages where both the width and depth are between the above thresholds are classified as general damage. Among them, the damage information includes damage category, damage position, damage size and damage degree, and the damage categories include cracks, spalling, corrosion, etc.

[0182] It should be noted that by fusing RGB images, infrared thermal images, and point cloud data, and combining visible light textures, thermal distributions, and three-dimensional geometric information, the structural state of the building can be reflected in multiple dimensions, enhancing the comprehensiveness and accuracy of monitoring. Using the PointNet neural network to automatically segment multi-dimensional point cloud data can efficiently disassemble the building into individual components (such as beams, columns, etc.), and accurately identify independent point clouds in combination with the structural characteristics of the building, providing a reliable basis for subsequent feature extraction and damage analysis, especially performing well in complex building environments. Conducting clustering analysis on the texture and block features in the component point cloud data can accurately locate the damage position and size of each component, and at the same time quantify the damage, providing data support for subsequent modeling.

[0183] In the embodiment, as Figure 5 shown, the visualization display module includes:

[0184] The damage information superposition unit is used to superpose and map the damage category, damage position, damage size, and damage degree on the three-dimensional model. The damage information is mapped to the three-dimensional model through color coding, and the color mapping is based on the following normalization formula;

[0185]

[0186] where s i is the normalized damage value, d min and d max are the minimum and maximum damage values respectively;

[0187] d i is the original damage value, that is, the specific damage measurement value before normalization;

[0188] The time series data display unit is used to display the damage evolution process of the building at different time points. Users can select different time points through the time axis to view the damage changes and dynamically track the expansion trend of the damage, providing multi-period data comparison;

[0189] The heat map generation and display unit is used to generate a dynamic heat map according to the damage severity, intuitively showing the damage risk of each area of the building. The color of the dynamic heat map is encoded according to the damage value, and the mapping formula is;

[0190] C(s i ) = C min +(C max -C min )·s i

[0191] where; C(S i ) is the normalized damage value S iThe color value mapped to the dynamic heat map, representing the color intensity of the damaged area in the heat map; C min The minimum color value, corresponding to the lower limit of the color intensity in the heat map; C max The maximum color value, corresponding to the upper limit of the color intensity in the heat map; S i The normalized damage value, usually ranging from [0, 1], representing the relative damage degree.

[0192] Through the dynamic heat map, users can quickly identify the areas where damage is concentrated, facilitating timely maintenance decisions.

[0193] In the embodiment, as Figure 6 shown, the prediction module includes:

[0194] The data analysis unit analyzes the historical multimodal data of the building structure and identifies the evolution trend of damage over time;

[0195] The damage trend prediction unit analyzes the evolution trend of damage over time through the damage prediction model, predicts the damage of the building structure, and obtains the prediction result. The prediction formula is: h t = O t * tanh(C t );

[0196] Among them, h t is the output of the damage prediction model, used to predict the damage state at a future time. O t is the output gate, and C t is the memory cell state, storing historical damage information and long-term dependencies;

[0197] The evaluation unit conducts a risk assessment on the prediction result to determine whether to trigger a warning signal.

[0198] In the embodiment, as Figure 7 shown, the evaluation report generation module includes:

[0199] The report automatic generation unit automatically generates a damage evaluation report according to the damage information and prediction result. The damage evaluation report includes a visual display of the damage trend, enabling users to intuitively view the damage expansion rate, trend changes, and predicted high-risk areas of the building.

[0200] The repair suggestion generation unit is used to automatically generate targeted repair suggestions according to the damage information and prediction result. For example, based on a preset rule library and combined with the safety standards of the building structure, it generates a repair plan with a higher priority. By analyzing the type, location, severity, and prediction result of the damage, it uses a rule engine to automatically recommend repair measures. Each type of damage corresponds to different repair suggestions and is displayed in the report.

[0201] In an embodiment, the system further includes a dynamic interaction module, which supports real-time interaction operations between the user and the 3D model and the display of dynamic information. For example, Figure 8 As shown, the dynamic interaction module includes:

[0202] A perspective transformation control unit for implementing perspective transformation of the user in the 3D model;

[0203] A timeline and temporal change display unit for displaying the change of damage data of the building at different times;

[0204] A dynamic heat map display unit for generating a real-time dynamic heat map according to the severity of the building damage.

[0205] For example, the user can control the perspective transformation of the 3D model through the interface, and can implement operations such as rotation, scaling, and translation. This function is realized through a perspective transformation matrix. To control the scaling of the model, to control the magnification or reduction of the model, and through the user's dragging operation, to realize the translation of the model, to control the movement of the model on the plane. All the user's interaction operations will generate a new perspective transformation matrix through matrix combination and act on the rendering process of the 3D model in real time. The user can also select different time nodes through the timeline to view the damage status of the building at different times and compare the changes in damage. At the same time, based on the prediction results of the temporal analysis module, the user can view the possible development trend of future damage. On the 3D model, the system dynamically displays the severity of the damage through a heat map, and the color mapping is associated with the degree of damage. The user can observe the real-time update of the heat map with the change of time or perspective during the interaction operation.

[0206] In an embodiment, the multi-modal data fusion unit includes the following steps:

[0207] Perform two-dimensional image fusion on the RGB image and the infrared thermal image to obtain a fused image containing RGB color and temperature information; specifically, it includes the following steps:

[0208] Based on the resolution of the RGB image, perform super-resolution reconstruction on the infrared thermal image using bilinear interpolation, and the resolution of the RGB image is equal to the resolution of the reconstructed infrared thermal image;

[0209] Unify the coordinate systems of the RGB image and the reconstructed infrared thermal image, and use a multi-scale fusion image fusion algorithm to process the coordinate systems of the RGB image and the reconstructed infrared thermal image to obtain a fused image containing RGB color and temperature information.

[0210] It should be noted that the coordinate systems of the RGB image and the reconstructed infrared thermal image are unified through manual registration.

[0211] Use the LBP local binary feature extraction operator to extract the texture features of the fused image;

[0212] The Sobel operator is used to extract the contour features of the fused image, and the contour features are converted into binary features;

[0213] Fusion is performed based on the texture features and binary features to form block features;

[0214] Based on the temperature information of the fused image, background temperature difference is used to obtain the temperature features of the image;

[0215] Combined with the relationship between the temperature features of the image and the damage depth, damage depth feature conversion is performed;

[0216] The converted damage depth features are synthesized with the fused image to form a multi-dimensional fused image; the multi-dimensional fused image is aligned and registered with the point cloud data, and the multi-dimensional information is mapped onto the 3D point cloud to obtain multi-dimensional spatial point cloud data. The multi-dimensional spatial point cloud data includes multi-dimensional information such as (x, y, z) spatial coordinate information, color, temperature, texture, contour, block, depth, etc.

[0217] In this embodiment, first, through the data acquisition and processing module, the physical state of the building and the changes in the surrounding environment can be monitored in real time. By regularly collecting multi-modal data, the system can provide dynamic health information of the building structure, and all the collected multi-modal data are processed and fused. Through preprocessing, data from different sensors can be cross-validated to eliminate redundancy and noise in the data, ensuring the accuracy and reliability of the final output data; through data fusion, the system can more comprehensively reflect the overall health state of the building, improving the effectiveness of detection. Using machine learning and data mining techniques, the system deeply analyzes and evaluates the fused data. By comparing historical data, the system can identify potential structural problems and their development trends. For example, the system can predict the future health state of the building, timely detect risks, and facilitate relevant personnel to take necessary preventive measures. The system sets an early warning threshold and continuously monitors the real-time data. If the detected data exceeds the safe range, the system will immediately issue an alarm to notify relevant personnel. This real-time early warning mechanism can timely respond to emergencies and ensure the safety of the building; through the user interface, users can easily access the health state of the building, historical monitoring data, and damage analysis reports. Through data visualization technology, users can intuitively understand the monitoring results, facilitating decision-making. In addition, the system regularly generates detailed health reports to provide a basis for subsequent maintenance and management.

[0218] The foregoing Figure 1All the various variations and specific examples of a building structure damage detection method in Embodiment 1 are equally applicable to a building structure damage detection system in this embodiment. Through the foregoing detailed description of a building structure damage detection method, those skilled in the art can clearly know the implementation method of a building structure damage detection system in this embodiment. Therefore, for the sake of brevity of the specification, it will not be elaborated here.

[0219] Embodiment 3

[0220] An embodiment of the present application provides a computer device, which includes a processor and a memory. At least one instruction or at least one program segment is stored in the memory, and the at least one instruction or the at least one program segment is loaded and executed by the processor to implement a building structure damage detection method as provided in the foregoing method embodiment.

[0221] Figure 9 The figure shows a schematic hardware structure diagram of a device for implementing a building structure damage detection method provided in an embodiment of the present application. The device may participate in forming or include a device or system provided in an embodiment of the present application. As Figure 9 shown, the computer device 10 may include one or more processors 1002 (the processor may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1004 for storing data, and a transmission device 1006 for communication functions. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 9 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the computer device 10 may further include more or fewer components than Figure 9 shown, or have a different configuration from Figure 9 shown.

[0222] It should be noted that the above-mentioned one or more processors and / or other data processing circuits are generally referred to as "data processing circuits" in this article. The data processing circuit may be embodied in whole or in part as software, hardware, firmware, or any combination thereof. In addition, the data processing circuit may be a single independent processing module, or be incorporated in whole or in part into any one of other elements in the computer device 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit is a processor control (such as the selection of a variable resistance terminal path connected to an interface).

[0223] The memory 1004 can be used to store software programs and modules of application software, such as the program instruction / data storage device corresponding to a building structure damage detection method in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 1004, that is, to implement the above-mentioned method. The memory 1004 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 1004 may further include a memory remotely disposed relative to the processor, and these remote memories can be connected to the computer device 10 through a network. Examples of the above-mentioned network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and combinations thereof.

[0224] The transmission device 1006 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by a communication provider of the computer device 10. In one instance, the transmission device 1006 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one instance, the transmission device 1006 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0225] The display can be, for example, a touch-screen liquid crystal display (LCD), which enables a user to interact with the user interface of the computer device 10 (or mobile device).

[0226] Embodiment 4

[0227] The embodiments of the present application further provide a computer-readable storage medium. The computer-readable storage medium can be disposed in a server to store at least one instruction or at least one segment of program related to implementing a building structure damage detection method in the method embodiments. The at least one instruction or the at least one segment of program is loaded and executed by the processor to implement a building structure damage detection method provided by the above method embodiments.

[0228] Optionally, in the embodiment, the above storage medium may be located in at least one of multiple network servers of a computer network. Optionally, in the embodiment, the above storage medium may include but is not limited to: USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical discs and other media that can store program codes.

[0229] Example 5

[0230] An embodiment of the present invention also provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes a building structure damage detection method provided in the above various alternative embodiments.

[0231] It should be noted that: the above sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above specific embodiments of the present application have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be executed in a different order from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0232] Each embodiment in the present application is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and the key point of each embodiment is to illustrate the differences from other embodiments. In particular, for the embodiments of the device, equipment, and storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments.

[0233] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above embodiments can be completed by hardware, or can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium, and the above-mentioned storage medium can be a read-only memory, a disk, an optical disc, etc.

[0234] Inspired by the above ideal embodiments of the present invention, through the above description, relevant workers can completely make various changes and modifications without departing from the technical idea of the present invention. The technical scope of the present invention is not limited to the content in the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A method for detecting damage to a building structure, characterized in that: The method comprises the following steps: S1, collecting multimodal data of a building structure and preprocessing the multimodal data; S2, construct a three-dimensional model of the building; S3, damage identification and classification of multimodal data are performed through the damage detection classification model to obtain damage information; S4, mapping the damage information on the three-dimensional model to perform visual monitoring of building damage; S5, analyzing the multimodal data through the damage prediction model, predicting the damage of the building structure, and obtaining the prediction result; S6, generating a damage assessment report based on the damage information and prediction results.

2. The building structure damage detection method according to claim 1, characterized in that: In the step S1, an image acquisition unit, an infrared thermal imager and a laser radar are used to acquire an RGB image, a thermal distribution image and point cloud data of the surface of the building structure, respectively, and the RGB image, the thermal distribution image and the point cloud data form the multimodal data; Preprocessing the multimodal data includes: Performing denoising processing on the RGB image and the thermal distribution image; The point cloud data is downsampled.

3. The building structure damage detection method according to claim 2, characterized in that: The step S3 comprises the following steps: S31, processing the denoised RGB image by an edge detection algorithm to extract damage edge features; S32, processing the damage edge feature to obtain the damage width; S33, fusing the RGB image, the infrared thermal image and the point cloud data to obtain multi-dimensional spatial point cloud data; S34, obtaining point cloud data of damage edge features, and obtaining the depth of the damage using surface fitting technology; S35, segmenting the multi-dimensional spatial point cloud data by using a PointNet neural network in combination with the building structure, and obtaining a point cloud data set of each structural component; S36, performing cluster analysis on texture features and block features in the point cloud data of each structural component to obtain damage locations and sizes of each structural component; S37, according to the damage position and size of each structural component, based on the normal direction and depth information of the point cloud, using voxel stretching to extend in the depth direction, expand in the horizontal direction, and merge the voxels to form a damaged spatial structure; S38, classifying the degree of damage according to the width and depth of the damage to obtain a classification result; The damage information includes damage type, damage location, damage size and damage degree.

4. The building structure damage detection method according to claim 1, characterized in that: The step S5 specifically comprises the following steps: Analyze historical multimodal data of building structures to identify damage evolution trends over time; The damage prediction model is used to analyze the evolution trend of the damage degree over time, and the damage prediction of the building structure is performed to obtain the prediction result. The prediction formula is: h t =O t *tanh(C t ); Among them, h t is the output of the damage prediction model, which is used to predict the damage state at future times. t is the output gate, C t is the memory unit state; Conduct risk assessment on the forecast results to determine whether to trigger early warning signals.

5. The building structure damage detection method according to claim 3, characterized in that: The step S33 specifically includes the following steps: S331, performing two-dimensional image fusion on the RGB image and the infrared thermal image to obtain a fused image containing RGB color and temperature information; S332, extracting texture features of the fused image using an LBP local binary feature extraction operator; S333, using the Sobel operator to extract the contour features of the fused image, and converting the contour features into binary features; S335, fusing the texture feature and the binary feature to form a block feature; S336, based on the temperature information of the fused image, using the background temperature for differentiation, to obtain the temperature feature of the image; S337, combining the relationship between the temperature feature of the image and the damage depth, performing damage depth feature conversion; S338, synthesizing the converted damage depth feature with the fused image to form a multi-dimensional fused image; S339, aligning and registering the multi-dimensional fused image with the point cloud data, mapping the multi-dimensional information onto the three-dimensional point cloud, and obtaining multi-dimensional spatial point cloud data.

6. The building structure damage detection method according to claim 1, characterized in that: The step S341 specifically includes the following steps: Based on the resolution of the RGB image, bilinear interpolation is used to perform super-resolution reconstruction on the infrared thermal image, and the resolution of the RGB image is equal to the resolution of the reconstructed infrared thermal image; The coordinate system of the RGB image and the coordinate system of the reconstructed infrared thermal image are unified, and the coordinate system of the RGB image and the reconstructed infrared thermal image are processed using a multi-scale fusion image fusion algorithm to obtain a fused image containing RGB color and temperature information.

7. The building structure damage detection method according to claim 1, characterized in that: The method also supports real-time interactive operations between users and three-dimensional models and display of dynamic information.

8. A building structure damage detection system, characterized in that: The detection system comprises: A data acquisition and processing module, which acquires multimodal data of the building structure and pre-processes the multimodal data; Building blocks to construct a three-dimensional model of the building; The damage detection and classification module uses the damage detection and classification model to identify and classify damages in multimodal data to obtain damage information; A visualization display module maps the damage information on the three-dimensional model to perform visual monitoring of building damage; The prediction module analyzes multimodal data through a damage prediction model, predicts damage to the building structure, and obtains prediction results; The assessment report generation module generates a damage assessment report based on the damage information and prediction results.

9. A computer device, characterized in that: include: processor; A memory for storing executable instructions; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the building structure damage detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the building structure damage detection method according to any one of claims 1 to 7.

Images