A rapid inspection method for brick-concrete building structures based on multimodal tightly coupled SLAM

Abstract

The present invention provides a method for rapid physical inspection of brick-concrete building structures using multimodal tightly coupled SLAM. Laser residuals, visual residuals, thermal imaging residuals, and IMU pre-integration residuals are introduced into the factor graph to construct a real-time positioning and mapping framework that can achieve high precision and strong robustness in low-light and texture-missing scenarios. During the mapping process, the system simultaneously runs dual-threshold recognition of crack width and temperature difference and wall inclination rate assessment to complete the risk identification and labeling of dangerous cracks and tilted walls in real time. A three-dimensional building model with attached semantic standards is generated to form an integrated closed-loop process that integrates "multimodal perception, high-precision mapping, crack and tilt risk assessment, and instant output warning", meeting the high-efficiency and high-precision structural inspection requirements of old brick-concrete buildings that "scan and judge danger". The present invention is suitable for scenarios such as urban renewal, dangerous building surveys, and structural safety monitoring of self-built houses in rural areas, significantly improving the efficiency and reliability of rapid building screening.

Description

Technical Field

[0001] The present invention relates to the technical field of building detection, and in particular to a rapid physical inspection method for brick-concrete building structures using multi-modal tightly coupled SLAM. Background Art

[0002] In recent years, handheld SLAM equipment (such as the Leica BLK2GO) has been widely used in the field of 3D architectural mapping. They scan existing buildings to form point clouds. However, due to the single sensor modality, positioning drift and missed defects are common in low-light scenarios, with monotonous textures or reflective facades. Furthermore, the lack of built-in detection algorithms makes it impossible to directly determine building safety issues on-site, and thus fails to meet the demand for rapid on-site inspections of old brick-concrete buildings. Furthermore, existing research has made progress in binocular vision fusion, crack identification using thermal and visible light image fusion, and tilt determination using point cloud fitting. However, a complete technical chain combining on-site scanning and positioning with rapid hazard detection and early warning has yet to be established. Summary of the Invention

[0003] (1) Technical issues to be resolved

[0004] Based on this, the present invention provides a rapid physical inspection method for brick-concrete building structures based on multi-modal tightly coupled SLAM to solve the problems mentioned in the background technology, such as the lack of infrared thermal image coordination in existing SLAM equipment, which leads to easy drift of positioning in weak light or mirror scenes and the need for manual correction; the crack recognition based on the fusion of thermal images and visible light images lacks the spatial correlation capability with the point cloud coordinate system, and thus cannot achieve real-time positioning and annotation of crack defects in three-dimensional building models; the current wall tilt judgment based on point cloud fitting lacks immediate warning capability, which loses the value of rapid on-site danger judgment.

[0005] (2) Technical solution

[0006] In order to achieve the above object, the present invention provides a rapid physical inspection method for brick-concrete building structures using multi-modal tightly coupled SLAM, comprising:

[0007] S1: Connect the LiDAR, IMU, RGB camera, fisheye camera, and infrared thermal camera in the handheld body to the same PPS pulse time base to unify the time system among all sensors;

[0008] S2: Hold the aircraft to obtain key frames, and convert the multi-channel sensor data into a unified aircraft coordinate system by reading the external and internal parameters of each sensor to achieve spatial alignment;

[0009] S3: The laser residuals obtained by the LiDAR, the visual residuals obtained by the RGB camera and the fisheye camera, the thermal image residuals obtained by the infrared thermal camera, and the pre-integrated residuals obtained by the IMU in the key frame are written into the factor graph. Based on SLAM, high-precision poses and incremental point clouds are optimized in real time on the environment map, and global correction is performed during loop closure detection.

[0010] S4: Read each key frame, automatically identify cracks, and obtain the center pixel coordinates of each crack , real physical width , temperature difference , Risk Marker and timestamp , stored in the crack sequence ;

[0011] S5: Map each crack from the image plane to the environment map to obtain a 3D crack sequence , Including the 3D coordinates of the center point of each crack , real physical width , temperature difference , Risk Marker , re-scan mark and timestamp ;

[0012] S6: Extract the nearly vertical wall of the building, then determine whether the wall is tilted, and generate a data set containing wall data information and tilt marks Wall sequence ;

[0013] S7: Displays and warns of cracks and tilted walls in the mobile view, and supports clicking to view and enter notes or take additional photos.

[0014] Furthermore, S3 specifically includes:

[0015] S301: Combine the LiDAR point cloud data of the first key frame with LOAM to extract edge feature points and plane feature points to generate a local environment map;

[0016] S302: Using the ICP or NDT algorithm, align the point cloud data collected by the LiDAR in the current key frame with the constructed environment map to calculate the position and posture of the aircraft;

[0017] S303: Using the ORB algorithm to extract visual features from the RGB image and fisheye image of the current key frame; extracting thermal imaging features from the infrared image of the current key frame based on the temperature gradient; then, matching the visual features and thermal imaging features of the current key frame with the corresponding features in the previous key frame image, calculating the reprojection error of the matching points, and obtaining a visual residual C and a thermal imaging residual H;

[0018] S304: Using high-frequency IMU data, pre-integrate the acceleration and angular velocity over a short period of time to calculate the inertia increment of the current key frame relative to the previous key frame; and use the error term between the inertia increment and the motion difference calculated from the estimated poses of the previous and next key frames as the IMU pre-integration residual I.

[0019] S305: Write the laser residual L, the visual residual C, the thermal image residual H and the IMU pre-integration residual I generated by the LiDAR point cloud registration into the factor graph, construct a nonlinear optimization problem with multimodal joint constraints, and obtain the optimal pose of the current key frame through comprehensive calculation by the Ceres optimizer. And incremental point cloud ; Then the posture Write to keyframe sequence , and the incremental point cloud Integrate into existing environment maps and update in real time.

[0020] Furthermore, S4 specifically includes:

[0021] S401: Read the RGB camera image and the infrared thermal imaging camera image in each key frame, reproject the infrared image into the RGB image coordinate system, fuse the temperature information of the infrared image at the pixel position of the RGB image, and correspondingly stitch them into a four-channel image, namely, the three color channels of R, G, and B + the temperature channel of the infrared thermal image, and perform mean-variance normalization processing on the four-channel image;

[0022] S402: Input the 4-channel image into the improved U-Net network, and finally output a crack distribution image, in which the value of each pixel in the crack distribution image is 0 or 1, where 0 indicates that there is no crack at the pixel position, and 1 indicates that there is a crack at the pixel position;

[0023] S403: extracting pixel outlines of cracks in the crack distribution image, counting and recording the widest points of all cracks and their horizontal pixel widths;

[0024] S404: Converting the horizontal pixel widths of all cracks into actual physical widths of the cracks;

[0025] S405: Get the average temperature at the widest point of all cracks , and the average temperature of the background area at the widest point , calculate the temperature difference ; Wherein, the background area refers to the area that does not belong to the crack in the widest 5×5 pixel area;

[0026] S406: If the actual physical width of the crack or temperature difference , then the crack is marked as a "dangerous crack", that is, the risk mark of the crack Set to 1; otherwise it is only used as a normal crack, that is, a risk marker for the crack Set to 0;

[0027] S407: The pixel coordinates of the center point of the widest point of each crack , real physical width , temperature difference , Risk Marker and timestamp Encapsulated as a crack record and stored in the crack sequence middle.

[0028] Furthermore, S5 specifically includes:

[0029] S501: Based on the crack sequence generated in step S4 , read the key frame sequence of step S3 Zhongtong Corresponding pose , as well as the external and internal parameters of each sensor in step S2, the pinhole imaging model is used to back-project the pixel coordinates of each crack to the camera coordinate system, and combined with the depth value obtained from the current frame LiDAR point cloud projected to the image plane, the three-dimensional coordinates of the center point pixel of the crack in the camera coordinate system are calculated ;

[0030] S502: The point cloud coordinates of the crack center point are obtained by mapping to the lidar coordinate system through coordinate transformation ;

[0031] S503: The point cloud coordinates of the crack center point , mapped to the body coordinate system through rigid transformation, and the position coordinates of the crack center point in the environment map are obtained ;Incremental point cloud Divided into voxel grids, each voxel unit is used to carry structural features and semantic labels;

[0032] S504: Find the voxel unit closest to the crack point in the voxel grid, and add the attribute information of the crack to the nearest voxel unit, including the three-dimensional coordinates of the center point of the crack , real physical width , temperature difference , timestamp , Risk Marker and rescan mark ,in The default value is 0, and the attribute information is added to the 3D crack sequence middle;

[0033] S505: In the process of mapping each crack from the image plane to the environment map, if it is found that the effective depth information of the pixel cannot be successfully projected from the LiDAR point cloud, , then the crack is marked with a fill-in mark It is marked as 1, and a prompt "The crack needs to be re-scanned" is displayed in the mobile view, guiding the user to go back to the crack area and retake the photo.

[0034] Furthermore, S6 specifically includes:

[0035] S601: Incremental point cloud for each key frame , the voxel filtering algorithm is used to pre-process the point cloud to remove outliers and high-frequency noise; then the RANSAC plane fitting algorithm is used to extract the near-vertical wall surfaces that meet the condition L1 in the point cloud and store them in the near-vertical wall sequence Among them, condition L1 includes: the wall area is greater than 0.5m², and the angle between the normal direction of the wall and the direction of gravity satisfy ;

[0036] S602: The orientation angles of all near-vertical walls in Enter the K-Means clustering algorithm for grouping, set the number of clusters k>4, and then select The main direction clusters; among them, the orientation angle of the nearly vertical wall The angle between the normal vector of the nearly vertical wall projected onto the horizontal ground and the reference coordinate axis; in all main direction clusters, sort them from large to small according to the plane area to obtain the wall set of all main direction clusters ;

[0037] S603: For the point cloud areas that do not meet the condition L1, a normal-guided Region-Growing algorithm is executed for wall compensation reconstruction; for small facets that are fragmented due to factors such as door openings and furniture occlusion, if the normal angle between the small facet and the adjacent facets is less than 5°, and the distance difference between the small facet and the fitting plane of the adjacent facets is less than 5cm, the small facet and the adjacent facets are merged into a continuous plane to obtain the compensated plane set. ,like The area of ​​any plane in the Any normal angle of the main direction cluster is less than the set threshold , then add the plane to the wall list ;

[0038] S604: Yes Point cloud of all walls in the image, extract the unit normal vectors of all walls and centroid , where the subscript is the wall number;

[0039] S605: The unit vector of the gravity direction As an absolute reference vertical direction, calculate The inclination angle of all walls in ,in , and convert it to get the wall inclination rate ,in ;

[0040] S606: If The inclination rate of the middle wall meets , then mark the wall as "inclined wall", that is, the inclination mark of the wall Set to 1; otherwise it will be treated as a normal wall. Set it to 0; then The walls and their tilt marks are stored in the wall sequence .

[0041] Furthermore, S7 specifically includes:

[0042] S701: The incremental point cloud obtained by the handheld body , the three-dimensional crack sequence and the wall sequence Pack and compress data and transmit it to the mobile terminal;

[0043] S702: The mobile terminal decompresses the received data and transforms the incremental point cloud based on the established point cloud coordinate transformation relationship. The position and perspective of the point cloud are updated and repositioned in real time, mapped to the scene coordinate system for visualization, ensuring that the 3D building model maintains a consistent spatial alignment with the real scene in the mobile view. In addition to real-time rendering of the incremental point cloud in the mobile view, Risk markers in and The tilt mark in , dangerous cracks are marked with red spheres at the corresponding locations. Cracks with a value of 1 are dangerous cracks, and are marked with yellow triangles to indicate tilted walls. Walls with a value of 1 are tilted walls. Users can click on the red sphere or yellow triangle in the mobile view to view relevant information. Clicking on the red sphere will read the actual physical width of the crack. , temperature difference and timestamp , click the yellow triangle to read the wall's inclination and timestamp , and can enter text notes and take photos on site; if in When it is 1, the mobile view interface will flash a prompt "Cracks need to be repaired" and highlight the crack areas that need to be repaired.

[0044] Furthermore, in S602 .

[0045] Furthermore, the threshold in S603 .

[0046] (3) Beneficial effects

[0047] From the above technical solution, it can be seen that the rapid physical inspection method of brick-concrete building structure using multi-modal tightly coupled SLAM proposed in the present invention has the following beneficial effects:

[0048] 1. By simultaneously introducing laser residuals, visual residuals, thermal image residuals, and IMU pre-integration residuals into the factor graph, high-precision real-time positioning and mapping are achieved even in low-light, mirror-like, or texture-sparse environments.

[0049] 2. It integrates multimodal data from LiDAR, IMU, RGB camera, fisheye camera and infrared thermal imaging camera, and simultaneously runs a dual-threshold recognition algorithm for crack width and temperature difference, as well as calculation of wall inclination, to achieve automatic hazard judgment, effectively reducing the risk of missed detection by a single sensor or a single algorithm.

[0050] 3. It can complete the three-dimensional positioning and visualization of two types of risks, cracks and tilted walls, in real time, and supports users to click to view relevant parameters and re-scan guidance, greatly improving the on-site decision-making efficiency and safety of inspection personnel.

[0051] 4. The present invention provides a rapid physical inspection method for brick-concrete building structures that integrates multi-modal synchronous acquisition, tightly coupled SLAM mapping, real-time danger assessment of cracks and tilted walls, and visual interaction. It takes into account real-time performance, accuracy and ease of use, and can meet the actual needs of large-scale rapid safety inspections of old buildings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0053] Figure 1 This is a flow chart of the rapid physical inspection method for brick-concrete building structures using multi-modal tightly coupled SLAM of the present invention;

[0054] Figure 2 Schematic diagram of the rapid physical inspection method of brick-concrete building structures using multi-modal tightly coupled SLAM of the present invention. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0056] like Figure 1-2 As shown, the present invention provides a rapid physical inspection method for brick-concrete building structures using multi-modal tightly coupled SLAM, comprising:

[0057] S1: Connect the LiDAR, IMU, RGB camera, fisheye camera, and infrared thermal camera in the handheld body to the same PPS pulse time base to unify the time system among all sensors;

[0058] The handheld device (hereafter referred to as the handheld device) integrates five sensors: a LiDAR (laser radar), an IMU (inertial measurement unit), an RGB camera, a fisheye camera, and an infrared thermal imaging camera. These five sensors are connected to the same PPS pulse timebase, enabling simultaneous exposure or scanning for all five sensors, ensuring synchronization of keyframes from each sensor. Each keyframe is then written with the same timestamp, known as a homologous timestamp.

[0059] S2: Hold the aircraft to obtain key frames, and convert the multi-channel sensor data into a unified aircraft coordinate system by reading the external and internal parameters of each sensor to achieve spatial alignment;

[0060] Complete the external parameter calibration between each sensor and the internal parameter calibration of each sensor, read the external parameter and internal parameter matrix of each sensor, obtain the data collected by each sensor under a unified clock based on the PPS pulse time base trigger, and transform the LiDAR point cloud coordinate system, IMU coordinate system, RGB camera coordinate system, fisheye camera coordinate system and infrared thermal imaging camera coordinate system into a unified body coordinate system.

[0061] That is to say, the key frame data obtained by the handheld body includes LiDAR point cloud data, IMU data, RGB camera images, fisheye camera images, infrared images of infrared thermal imaging cameras and body posture at the same timestamp.

[0062] S3: The laser residuals obtained by the LiDAR, the visual residuals obtained by the RGB camera and the fisheye camera, the thermal image residuals obtained by the infrared thermal camera, and the pre-integrated residuals obtained by the IMU in the key frame are written into the factor graph. Based on SLAM, high-precision pose and incremental point cloud are optimized in real time on the environment map, and global correction is performed during loop closure detection. Specifically, it includes:

[0063] S301: The LiDAR point cloud data of the first keyframe is combined with LOAM (Laser Odometry and Mapping, a typical laser SLAM algorithm) to extract edge feature points and plane feature points to generate a local environment map (specifically a 3D building model in this embodiment) for feature matching and pose registration of subsequent keyframe point clouds;

[0064] S302: Using the ICP or NDT algorithm, align the point cloud data collected by the LiDAR in the current key frame with the constructed environment map to calculate the position and posture of the aircraft;

[0065] S303: Using the ORB algorithm, extract visual features from the RGB image and fisheye image of the current keyframe. Extract thermal imaging features from the infrared image of the current keyframe based on the temperature gradient. Then, match these features of the current keyframe (including the visual features of the RGB image and fisheye image, and the thermal imaging features of the infrared image) with the corresponding features in the previous keyframe image. Calculate the reprojection error of the matching points to obtain the visual residual C and the thermal imaging residual H.

[0066] In this paper, cameras include RGB cameras and fisheye cameras; images include RGB images, fisheye images and infrared images; and visual images include RGB images and fisheye images.

[0067] S304: Using high-frequency IMU data, pre-integrate the acceleration and angular velocity over a short period of time to calculate the inertia increment of the current keyframe relative to the previous keyframe. This is used to assist in feature matching of subsequent images and LiDAR point cloud registration, improving registration efficiency and convergence stability. The error term between the inertia increment and the motion difference calculated from the estimated poses of the previous and next keyframes is used as the IMU pre-integration residual I, which is written into the factor graph for optimization.

[0068] S305: Write the laser residual L, the visual residual C, the thermal image residual H and the IMU pre-integration residual I generated by the LiDAR point cloud registration into the factor graph, construct a nonlinear optimization problem with multimodal joint constraints, and obtain the optimal pose of the current key frame through comprehensive calculation by the Ceres optimizer. And incremental point cloud (new point cloud data) ; Then the posture Write to keyframe sequence , and the incremental point cloud Integrate into existing environment maps to achieve real-time updates;

[0069] Among them, the key frame sequence The timestamp of the key frame is recorded , LiDAR point cloud data, IMU data, RGB camera images, fisheye camera images, infrared images from infrared thermal imaging cameras, and body pose.

[0070] S306: If a feature highly similar to the current key frame image is detected in the historical key frame image, and the judgment standard is similarity > 0.75, it is determined that a closed loop occurs. In addition to incorporating the current key frame into the factor graph to construct the constraint relationship, the posture of all key frames will be re-adjusted as a whole to obtain the corrected posture. ,Will cover The original position , in order to effectively offset the drift accumulation and achieve global correction.

[0071] S4: Read each key frame, automatically identify cracks, and obtain the center pixel coordinates of each crack , real physical width , temperature difference , Risk Marker and timestamp , stored in the crack sequence ; Specifically include:

[0072] S401: Read the RGB camera image and the infrared thermal imaging camera image in each key frame, reproject the infrared image into the RGB image coordinate system, fuse the temperature information of the infrared image at the pixel position of the RGB image, and correspondingly stitch them into a 4-channel image (i.e., the three color channels of R, G, and B + the temperature channel of the infrared thermal image), and perform mean-variance normalization processing on the 4-channel image;

[0073] S402: Input the 4-channel image into the improved U-Net network, and finally output a crack distribution image (i.e., crack mask). The value of each pixel in the crack distribution image is 0 or 1, where 0 indicates that there is no crack at the pixel location, and 1 indicates that there is a crack at the pixel location;

[0074] In this embodiment, the improved U-Net network adopts lightweight MobileV2 as the encoder and adds a multi-scale attention mechanism in the decoding stage; and the improved U-Net network is trained.

[0075] S403: extracting pixel outlines of cracks in the crack distribution image, counting and recording the widest points of all cracks and their horizontal pixel widths;

[0076] The specific operation is: count the length of each crack's continuous pixel value of 1 in the direction orthogonal to its direction, select the maximum length of the line segment in the crack as the horizontal pixel width of the widest part of the crack; the line segment with the largest length value is the widest part of the crack.

[0077] S404: Converting the horizontal pixel widths of all cracks into actual physical widths of the cracks;

[0078] Considering the distance between the handheld body and the wall during the acquisition process It is not fixed. To ensure the accuracy of the size calculation, it is necessary to geometrically convert the horizontal pixel width at the widest part of the crack based on the pinhole imaging model. Combined with the camera's in-camera pinhole imaging model obtained in step S2 and the depth information corresponding to the pixel (obtained by the current keyframe LiDAR point cloud projection), which can convert the horizontal pixel width at the widest point of the crack into the actual physical width of the crack ;

[0079] S405: Get the average temperature at the widest point of all cracks , and the average temperature of the background area at the widest point , calculate the temperature difference ; Wherein, the background area refers to the area that does not belong to the crack in the widest 5×5 pixel area;

[0080] S406: If the actual physical width of the crack or temperature difference , then the crack is marked as a "dangerous crack", that is, the risk mark of the crack Set to 1; otherwise it is only used as a normal crack, that is, a risk marker for the crack Set to 0;

[0081] S407: The pixel coordinates of the center point of the widest point of each crack , real physical width , temperature difference , Risk Marker and timestamp Encapsulated as a crack record and stored in the crack sequence It is used for subsequent three-dimensional projection and positioning.

[0082] S5: Map each crack from the image plane to the environment map to obtain a three-dimensional crack sequence , Including the three-dimensional coordinates of the center point of each crack , real physical width , temperature difference , Risk Marker , re-scan mark and timestamp ;

[0083] Based on the crack sequence generated in step S4 , the key frame sequence of step S3 Zhongtong Corresponding body posture , as well as the external and internal parameters of each sensor in step S2, project each crack from the pixel coordinate system of the image to the real-time three-dimensional coordinate system where the lidar point cloud is located. After completing the projection transformation, the three-dimensional coordinates of the center point of each crack are obtained , real physical width , temperature difference , Risk Marker , re-scan mark and timestamp Attribute information such as 3D crack sequence It is used for subsequent structural risk analysis and spatial positioning visualization. S5 specifically includes:

[0084] S501: Based on the crack sequence generated in step S4 , read the key frame sequence of step S3 Zhongtong Corresponding pose , as well as the external and internal parameters of each sensor in step S2, the pinhole imaging model is used to back-project the pixel coordinates of each crack to the camera coordinate system, and combined with the depth value obtained from the current frame LiDAR point cloud projected to the image plane, the three-dimensional coordinates of the center point pixel of the crack in the camera coordinate system are calculated ;

[0085] S502: The point cloud coordinates of the crack center point are obtained by mapping to the lidar coordinate system through coordinate transformation , to ensure that the cracks and laser point clouds are in a unified geometric reference frame, which is convenient for subsequent fusion processing and spatial analysis;

[0086] S503: The point cloud coordinates of the crack center point , mapped to the body coordinate system through rigid transformation, and the position coordinates of the crack center point in three-dimensional space (environment map) are obtained ;Incremental point cloud Divided into voxel grids, each voxel unit is used to carry structural features and semantic labels;

[0087] S504: Find the voxel unit closest to the crack point (the point of the crack in the point cloud) in the voxel grid, and add the attribute information of the crack to the nearest voxel unit, including the three-dimensional coordinates of the center point of the crack , real physical width , temperature difference , timestamp , Risk Marker and rescan mark (default is 0) and adds the attribute information to the 3D crack sequence middle;

[0088] S505: In the process of mapping each crack from the image plane to the environment map, if it is found that the effective depth information of the pixel cannot be successfully projected from the LiDAR point cloud, , then the crack is marked with a fill-in mark It is marked as 1, and a prompt "The crack needs to be re-scanned" is displayed in the mobile view, guiding the user to go back to the crack area and retake the photo.

[0089] S6: Extract the nearly vertical wall of the building, then determine whether the wall is tilted, and generate a data set containing wall data information and tilt marks Wall sequence ; Specifically include:

[0090] S601: Incremental point cloud for each key frame , the voxel filtering algorithm is used to pre-process the point cloud to remove outliers and high-frequency noise; then the RANSAC plane fitting algorithm is used to extract the near-vertical wall surfaces that meet the condition L1 in the point cloud and store them in the near-vertical wall sequence Among them, condition L1 includes: the wall area is greater than 0.5m², and the angle between the normal direction of the wall and the direction of gravity satisfy ;

[0091] S602: The orientation angles of all near-vertical walls in Input K-Means clustering algorithm to group. To avoid omission of initial classification, set the number of clusters k>4, and then select The main direction clusters (usually the main facade, side walls and interior partition walls); among them, the orientation angle of the nearly vertical wall The angle between the normal vector of the nearly vertical wall projected onto the horizontal ground and the reference coordinate axis; in all main direction clusters, sort them from large to small according to the plane area to obtain the wall set of all main direction clusters ;

[0092] In this embodiment, , The number of columns is determined by the complexity of the building.

[0093] S603: For the point cloud areas that do not meet the condition L1, a normal-guided Region-Growing algorithm is executed for wall compensation reconstruction; for small facets that are fragmented due to factors such as door openings and furniture occlusion, if the normal angle between the small facet and the adjacent facets is less than 5°, and the distance difference between the small facet and the fitting plane of the adjacent facets is less than 5cm, the small facet and the adjacent facets are merged into a continuous plane to obtain the compensated plane set. ,like The area of ​​any plane in the Any normal angle of the main direction cluster is less than the set threshold (such as 10°), then add the plane to the wall list , used to construct the complete set of walls required for structural integrity analysis;

[0094] S604: Yes Point cloud of all walls in the image, extract the unit normal vectors of all walls and centroid (Subscript is the wall number, the same below), providing basic geometric parameters for subsequent wall inclination calculation and structural stability assessment;

[0095] S605: The unit vector of the gravity direction As an absolute reference vertical direction, calculate The inclination angle of all walls in ,in , and convert it to get the wall inclination rate ,in ;

[0096] S606: If The inclination rate of the middle wall meets , then mark the wall as "inclined wall", that is, the inclination mark of the wall Set to 1; otherwise it will be treated as a normal wall. Set it to 0; then The walls and their tilt marks are stored in the wall sequence .

[0097] S7: Display and warn of cracks and tilted walls in the mobile view, and support clicking to view and enter notes or take additional photos; specifically:

[0098] S701: The incremental point cloud obtained by the handheld body , the three-dimensional crack sequence and the wall sequence Pack and compress data and transmit it to the mobile terminal;

[0099] Among them, the point cloud data uses the octree structure for spatial hierarchical compression to reduce redundant points and improve data transmission efficiency; the three-dimensional crack sequence and wall sequence The data is encoded using a lightweight and fast compression algorithm (LZ4) to reduce the overall data packet size.

[0100] S702: The mobile terminal decompresses the received data and transforms the incremental point cloud based on the established point cloud coordinate transformation relationship. The position and perspective of the point cloud are updated and repositioned in real time, mapped to the scene coordinate system for visualization, ensuring that the 3D building model maintains a consistent spatial alignment with the real scene in the mobile view. In addition to real-time rendering of the incremental point cloud in the mobile view, Risk markers in and The tilt mark in , dangerous cracks are marked with red balls at the corresponding locations ( is 1), and the inclined wall is marked with a yellow triangle. 1), to highlight its spatial location and extent; users can click the red sphere or yellow triangle in the mobile view to view relevant information (for example, click the red sphere to read the actual physical width of the crack , temperature difference and timestamp , click the yellow triangle to read the wall's inclination and timestamp ), and can enter text notes and take additional photos on site; if in When it is 1, the mobile view interface will flash a prompt "Cracks need to be repaired" and highlight the crack areas that need to be repaired.

[0101] The present invention provides a rapid physical inspection method for brick-concrete building structures that integrates multimodal synchronous acquisition, tightly coupled SLAM mapping, real-time danger assessment of cracks and tilted walls, and visual interaction. It takes into account real-time performance, accuracy, and ease of use, and can meet the actual needs of large-scale rapid safety inspections of old buildings.

[0102] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention.

Claims

1. A multi-modal tightly coupled SLAM rapid physical inspection method for brick-concrete building structures, characterized by: include: S1: Connect the LiDAR, IMU, RGB camera, fisheye camera, and infrared thermal camera in the handheld body to the same PPS pulse time base to unify the time system among all sensors; S2: Hold the aircraft to obtain key frames, and convert the multi-channel sensor data into a unified aircraft coordinate system by reading the external and internal parameters of each sensor to achieve spatial alignment; S3: The laser residuals obtained by the LiDAR, the visual residuals obtained by the RGB camera and the fisheye camera, the thermal image residuals obtained by the infrared thermal camera, and the pre-integrated residuals obtained by the IMU in the key frame are written into the factor graph. Based on SLAM, high-precision poses and incremental point clouds are optimized in real time on the environment map, and global correction is performed during loop closure detection. S4: Read each key frame, automatically identify cracks, and obtain the center pixel coordinates of each crack , real physical width , temperature difference , Risk Marker and timestamp , stored in the crack sequence ; S5: Map each crack from the image plane to the environment map to obtain a three-dimensional crack sequence , Including the three-dimensional coordinates of the center point of each crack , real physical width , temperature difference , Risk Marker , re-scan mark and timestamp ; S6: Extract the nearly vertical wall of the building, then determine whether the wall is tilted, and generate a data set containing wall data information and tilt marks Wall sequence ; S7: Displays and warns of cracks and tilted walls in the mobile view, and supports clicking to view and enter notes or take additional photos.

2. The method according to claim 1, characterized in that S3 specifically includes: S301: Combine the LiDAR point cloud data of the first key frame with LOAM to extract edge feature points and plane feature points to generate a local environment map; S302: Using the ICP or NDT algorithm, align the point cloud data collected by the LiDAR in the current key frame with the constructed environment map to calculate the position and pose of the aircraft; S303: Using the ORB algorithm to extract visual features from the RGB image and fisheye image of the current key frame; extracting thermal imaging features from the infrared image of the current key frame based on the temperature gradient; then, matching the visual features and thermal imaging features of the current key frame with the corresponding features in the previous key frame image, calculating the reprojection error of the matching points, and obtaining a visual residual C and a thermal imaging residual H; S304: Using high-frequency IMU data, pre-integrate the acceleration and angular velocity over a short period of time to calculate the inertia increment of the current key frame relative to the previous key frame; and use the error term between the inertia increment and the motion difference calculated from the estimated poses of the previous and next key frames as the IMU pre-integration residual I. S305: Write the laser residual L, the visual residual C, the thermal image residual H and the IMU pre-integration residual I generated by the LiDAR point cloud registration into the factor graph, construct a nonlinear optimization problem with multimodal joint constraints, and obtain the optimal pose of the current key frame through comprehensive calculation by the Ceres optimizer. And incremental point cloud ; Then the posture Write to keyframe sequence , and the incremental point cloud Integrate into existing environment maps and update in real time.

3. The method according to claim 2, characterized in that S4 specifically includes: S401: Read the RGB camera image and the infrared thermal imaging camera image in each key frame, reproject the infrared image into the RGB image coordinate system, fuse the temperature information of the infrared image at the pixel position of the RGB image, and correspondingly stitch them into a four-channel image, namely, the three color channels of R, G, and B + the temperature channel of the infrared thermal image, and perform mean-variance normalization processing on the four-channel image; S402: Input the 4-channel image into the improved U-Net network, and finally output a crack distribution image, in which the value of each pixel in the crack distribution image is 0 or 1, where 0 indicates that there is no crack at the pixel position, and 1 indicates that there is a crack at the pixel position; S403: extracting pixel outlines of cracks in the crack distribution image, counting and recording the widest points of all cracks and their horizontal pixel widths; S404: Converting the horizontal pixel widths of all cracks into actual physical widths of the cracks; S405: Get the average temperature at the widest point of all cracks , and the average temperature of the background area at the widest point , calculate the temperature difference ; Wherein, the background area refers to the area that does not belong to the crack in the widest 5×5 pixel area; S406: If the actual physical width of the crack or temperature difference , then the crack is marked as a "dangerous crack", that is, the risk mark of the crack Set to 1; otherwise it is only used as a normal crack, that is, a risk marker for the crack Set to 0; S407: The pixel coordinates of the center point of the widest point of each crack , real physical width , temperature difference , Risk Marker and timestamp Encapsulated as a crack record and stored in the crack sequence middle.

4. The method according to claim 3, characterized in that S5 specifically includes: S501: Based on the crack sequence generated in step S4 , read the key frame sequence of step S3 Zhongtong Corresponding pose , as well as the external and internal parameters of each sensor in step S2, the pinhole imaging model is used to back-project the pixel coordinates of each crack to the camera coordinate system, and combined with the depth value obtained from the current frame LiDAR point cloud projected to the image plane, the three-dimensional coordinates of the center point pixel of the crack in the camera coordinate system are calculated ; S502: The point cloud coordinates of the crack center point are obtained by mapping to the lidar coordinate system through coordinate transformation ; S503: The point cloud coordinates of the crack center point , mapped to the body coordinate system through rigid transformation, and the position coordinates of the crack center point in the environment map are obtained ;Incremental point cloud Divided into voxel grids, each voxel unit is used to carry structural features and semantic labels; S504: Find the voxel unit closest to the crack point in the voxel grid, and add the attribute information of the crack to the nearest voxel unit, including the three-dimensional coordinates of the center point of the crack , real physical width , temperature difference , timestamp , Risk Marker and rescan mark ,in The default value is 0, and the attribute information is added to the 3D crack sequence middle; S505: In the process of mapping each crack from the image plane to the environment map, if it is found that the effective depth information of the pixel cannot be successfully projected from the LiDAR point cloud, , then the crack is marked with a fill-in mark Mark it as 1, and prompt "Cracks need to be re-scanned" in the mobile view to guide the user to go back to the crack area and retake the photo.

5. The method according to claim 4, characterized in that S6 specifically includes: S601: Incremental point cloud for each key frame , the voxel filtering algorithm is used to pre-process the point cloud to remove outliers and high-frequency noise; then the RANSAC plane fitting algorithm is used to extract the near-vertical wall surfaces that meet the condition L1 in the point cloud and store them in the near-vertical wall sequence Among them, condition L1 includes: the wall area is greater than 0.5m², and the angle between the normal direction of the wall and the direction of gravity satisfy ; S602: The orientation angles of all near-vertical walls in Enter the K-Means clustering algorithm for grouping, set the number of clusters k > 4, and then select The main direction clusters; among them, the orientation angle of the nearly vertical wall The angle between the normal vector of the nearly vertical wall projected onto the horizontal ground and the reference coordinate axis; in all main direction clusters, sort them from large to small according to the plane area to obtain the wall set of all main direction clusters ; S603: For the point cloud areas that do not meet the condition L1, a normal-guided Region-Growing algorithm is executed for wall compensation reconstruction; for small facets that are fragmented due to factors such as door openings and furniture occlusion, if the normal angle between the small facet and the adjacent facets is less than 5°, and the distance difference between the small facet and the fitting plane of the adjacent facets is less than 5cm, the small facet and the adjacent facets are merged into a continuous plane to obtain the compensated plane set. ,like The area of ​​any plane in the Any normal angle of the main direction cluster is less than the set threshold , then add the plane to the wall list ; S604: Yes Point cloud of all walls in the image, extract the unit normal vectors of all walls and centroid , where the subscript is the wall number; S605: The unit vector of the gravity direction As an absolute reference vertical direction, calculate The inclination angle of all walls in ,in , and convert it to get the wall inclination rate ,in ; S606: If The inclination rate of the middle wall meets , then mark the wall as "inclined wall", that is, the inclination mark of the wall Set to 1; otherwise it will be treated as a normal wall. Set it to 0; then The walls and their tilt marks are stored in the wall sequence .

6. The method according to claim 5, characterized in that S7 specifically includes: S701: The incremental point cloud obtained by the handheld body , the three-dimensional crack sequence and the wall sequence Pack and compress data and transmit it to the mobile terminal; S702: The mobile terminal decompresses the received data and transforms the incremental point cloud based on the established point cloud coordinate transformation relationship. The position and perspective of the point cloud are updated and repositioned in real time, mapped to the scene coordinate system for visualization, ensuring that the 3D building model maintains a consistent spatial alignment with the real scene in the mobile view. In addition to real-time rendering of the incremental point cloud in the mobile view, Risk markers in and The tilt mark in , dangerous cracks are marked with red spheres at the corresponding locations. Cracks with a value of 1 are dangerous cracks, and are marked with yellow triangles to indicate tilted walls. Walls with a value of 1 are tilted walls. Users can click on the red sphere or yellow triangle in the mobile view to view relevant information. Clicking on the red sphere will read the actual physical width of the crack. , temperature difference and timestamp , click the yellow triangle to read the wall's inclination and timestamp , and can enter text notes and take photos on site; if in When it is 1, the mobile view interface will flash the prompt "Cracks need to be repaired" and highlight the crack areas that need to be repaired.

7. The method according to claim 5, characterized in that S602 .

8. The method according to claim 5, characterized in that S603 medium threshold .

Images