Rapid diagnosis method for brick-concrete mixed structure buildings using multimodal tightly coupled SLAM

Abstract

This invention provides a rapid diagnosis method for brick-and-concrete mixed-structure buildings based on multimodal tightly coupled SLAM. By incorporating laser residuals, visual residuals, thermal image residuals, and IMU pre-integration residuals into a factor graph, it establishes a real-time localization and map construction framework with high accuracy and robustness even in low-light and texture-depleted scenes. During the construction process, the system simultaneously performs dual-threshold classification of crack width and temperature difference and wall slope evaluation to complete real-time risk assessment and annotation of dangerous cracks and sloped walls. Finally, it generates a 3D building model with semantic annotations, forming an integrated closed-loop process that integrates multimodal perceptual processing, high-precision map construction, crack and slope risk assessment, and immediate output warnings. This meets the needs for high-time and high-precision structural diagnosis of dilapidated brick-and-concrete mixed-structure buildings, enabling immediate risk assessment through scanning. This method is applicable to urban redevelopment, dangerous building inspections, and structural safety monitoring of rural self-built housing, significantly improving the efficiency and reliability of rapid building screening.

Description

[Technical Field]

[0001] The present invention relates to the field of building inspection technology, and more particularly to a method for rapid diagnosis of brick-concrete mixed structure buildings using multimodal tightly coupled SLAM. [Background technology]

[0002] Already 20% of existing homes in urban areas in Japan are more than 30 years old, and this percentage is predicted to increase to approximately 80% by 2040. In particular, the large-scale brick and concrete mixed structure buildings constructed in the 1980s and 1990s were built based on the relatively low seismic design standards of the time, and therefore have a clear lack of seismic performance compared to current standards, increasing safety risks and making it necessary to eliminate potential dangers through regular inspections.

[0003] In recent years, portable SLAM devices (e.g., Leica® BLK2GO) have been widely used in the field of architectural 3D surveying, and are capable of scanning existing buildings to generate point clouds. However, because the sensor uses a single modality, position estimation drift and defects are often overlooked in scenes with low light, monotonous textures, or reflective exterior walls. Furthermore, since the device does not have built-in relevant detection algorithms, it is not possible to directly determine the safety of buildings on site, and it cannot meet the rapid diagnosis needs of "on-site immediate risk assessment" for dilapidated brick and concrete mixed structure buildings.

[0004] Furthermore, while existing research has made progress in areas such as "stereo vision fusion," "crack identification by fusing thermal and visible images," and "slope determination by point cloud fitting," the complete technical process leading to "on-site scanning positioning + rapid risk determination" has not yet been established. Summary of the Invention [Problem to be solved by the invention]

[0005] Based on this background, the present invention provides a rapid diagnosis method for brick-concrete mixed structure buildings using multimodal tightly coupled SLAM. The present invention solves the following problems pointed out in the background art. In other words, because conventional SLAM devices lack the ability to collaborate with infrared thermal images, position estimation tends to deviate significantly in low-light environments or environments with specular reflections, requiring manual correction. Furthermore, crack detection that combines thermal and visible images cannot spatially match with a point cloud coordinate system, making it impossible to identify and display crack locations in real time on a 3D architectural model. Furthermore, wall inclination determination using point cloud fitting lacks the ability to notify immediate danger, making it impossible to achieve the goal of quickly assessing danger on-site. [Means for solving the problem]

[0006] To achieve the above object, the present invention provides a method for rapid diagnosis of brick-concrete mixed structure buildings using multimodal tightly coupled SLAM, which includes the following steps:

[0007] S1: Connect the LiDAR, IMU, RGB camera, fisheye camera, and infrared thermal camera mounted on a mobile device to the same PPS pulse time base to synchronize the time between the sensors. S2: The handheld device acquires the keyframe, reads the internal and external parameters of each sensor, and transforms the multi-sensor data into the device coordinate system for spatial alignment. S3: For each keyframe, the laser residual from LiDAR, the visual residual from the RGB camera and fisheye camera, the thermal image residual from the infrared thermal camera, and the pre-integration residual from the IMU are input into a factor graph, and based on SLAM, high-precision pose and incremental point cloud are optimized in real time on the environment map, and overall re-optimization is performed when the loop is closed. S4: Read each key frame, automatically identify cracks, and obtain the central pixel point coordinates C(u,v), actual physical width w, temperature difference ΔT, danger flag DangerFlag, and time identification TimeID for each crack, and save them in the crack list Crack_list. S5: Project each crack from the image plane onto the environment map to obtain a 3D crack list Crack3D_list. Crack3D_list includes the three-dimensional coordinates (X, Y, Z) of the center of each crack, the actual physical width w, the temperature difference ΔT, the danger flag DangerFlag, the rescan indicator ProjStatus, and the time identification TimeID. S6: Extract the approximately vertical wall surfaces of the building, then determine whether or not the wall surfaces are inclined, and generate a wall surface list TiltList including the wall surface data and the inclination flag TiltFlag. S7: Cracks and sloping walls are displayed and a warning is issued on the mobile device screen. Clicking the screen allows the user to check related parameters, enter notes, or take additional photos.

[0008] Furthermore, S3 specifically includes the following processing: S301: For the LiDAR point cloud data of the first keyframe, edge feature points and plane feature points are extracted using LOAM (LiDAR Odometry and Mapping) to generate a local environment map. S302: Using the ICP (Iterative Closest Point) or NDT (Normal Distribution Transform) algorithm, the LiDAR point cloud data of the current keyframe is aligned with the pre-built environment map to calculate the device pose. S303: Extract visual features from the RGB image and fisheye image of the current keyframe using the ORB algorithm, and extract thermal image features from the infrared thermal image based on the temperature gradient. Then, match these features with the corresponding features of the previous keyframe, calculate the reprojection error of the corresponding points, and obtain the visual residual C and the thermal image residual H. S304: Pre-integrate based on the IMU data, estimate the cumulative attitude change based on the acceleration and angular velocity measurements, and obtain the IMU pre-integration residual I. S305: The laser residual L, visual residual C, thermal image residual H, and IMU pre-integration residual I obtained by aligning the LiDAR point cloud data are simultaneously written into a factor graph to construct a nonlinear optimization problem with multi-modal coupling constraints. Then, a Ceres optimizer performs joint calculations to find the optimal pose P of the current keyframe. t and incremental point cloud ΔP are obtained. Next, the posture P t is written into the keyframe sequence Pose_list, and the incremental point cloud ΔP is merged into the existing environment map to achieve real-time update.

[0009] Furthermore, S4 specifically includes the following processing. S401: Read the RGB camera image and infrared thermal camera image of each key frame, re-project the infrared image onto the coordinate system of the RGB image, and fuse the temperature information of the infrared image at each pixel position of the RGB image to generate a four-channel image with three color channels of R, G, and B and a temperature channel of the infrared thermal image, and perform mean-variance normalization processing on the four-channel image. S402: The four-channel image is input into an improved U-Net network, and finally a crack distribution image is output. Each pixel value in the crack distribution image is either 0 or 1, where 0 indicates that there is no crack at that pixel position, and 1 indicates that there is a crack at that pixel position. S403: Extract pixel contours of cracks in the crack distribution image, and statistically record the widest part of all cracks and their lateral pixel widths. S404: The horizontal pixel widths of all cracks are converted to the actual physical crack widths. S405: Average temperature T of the widest part of all cracks c and the average temperature Tb of the widest background area is obtained, and the temperature difference ΔT = T c -T b Here, the background area refers to the widest 5×5 pixel area that does not belong to the crack. S406: If the actual physical width of the crack is w≧1 mm or the temperature difference ΔT≧1°C, mark the crack as a "danger crack", i.e., set the danger flag DangerFlag of the crack to 1. Otherwise, treat it as a normal crack and set the danger flag DangerFlag of the crack to 0. S407: A crack record including the central pixel coordinates C(u,v) of the widest part of each crack, the actual physical width w, the temperature difference ΔT, the danger flag DangerFlag, and the time identification TimeID is created and saved in a crack list Crack_list.

[0010] Furthermore, S5 specifically includes the following processing: S501: Based on the crack list Crack_list generated in S4, the pose P in the key frame sequence Pose_list in S3 corresponding to the Time ID of the record is calculated. t Further, based on the external and internal parameters of each sensor in S2, the pixel coordinates of each crack are back-projected to the camera coordinate system using a pinhole camera model, and combined with the depth value obtained by projecting the LiDAR point cloud data of the current key frame onto the image plane, the center pixel of the crack is calculated as a three-dimensional coordinate (X cam ,Y cam ,Z cam ) is calculated. S502: (X cam ,Y cam ,Z cam ) is projected to the LiDAR coordinate system by coordinate transformation, and the point cloud coordinates of the crack center point (X lid ,Y lid ,Z lid ) is obtained. S503: Point cloud coordinates of the crack center point (X lid ,Y lid ,Z lid ) is projected to the device coordinate system by coordinate transformation to obtain the position coordinates (X, Y, Z) of the crack center point in the environment map. In addition, the incremental point cloud ΔP is divided into a voxel grid, and each voxel unit is used for structural features and labeling. S504: The voxel grid is searched for the voxel unit closest to the crack, and crack attribute information is added to the closest voxel unit. The attribute information includes the three-dimensional coordinates (X, Y, Z) of the crack center point, the actual physical width w, the temperature difference ΔT, the time identification TimeID, the danger flag DangerFlag, and the rescan indicator ProjStatus. Here, the default value of ProjStatus is set to 0, and the attribute information is added to the three-dimensional crack list Crack3D_list. S505: In the process of projecting the crack from the image plane to the environment map, the LiDAR point cloud data calculates the effective depth information Z of the pixel. cam If the crack cannot be projected, the rescan indicator ProjStatus for the crack is set to 1, and a "crack rescan instruction" is displayed in the mobile terminal image, prompting the user to rescan the crack area.

[0011] Furthermore, S6 specifically includes the following processing. S601: Pre-processing is performed on the incremental point cloud ΔP of each keyframe using a voxel filter operation to remove outlier points and high-frequency noise. Next, the RANSAC plane detection algorithm is used to extract approximately vertical walls that satisfy condition L1 from the point cloud, and these approximately vertical walls are saved in the vertical wall set PlaneSet0. Here, condition L1 includes the following: wall area > 0.5 m 2 , and the angle between the normal direction to the wall and the direction of gravity satisfies 70°≦α≦110°. S602: The normal direction angle β of all nearly vertical wall surfaces in PlaneSet0 is input into K-Means clustering, and the number of clusters k is set to be greater than 4 for classification. Then, n major direction clusters are selected based on empirical rules and spatial structure features. Here, the normal direction angle β of a nearly vertical wall surface is the angle formed when the normal direction of the nearly vertical wall surface is projected onto the horizontal ground direction and the gravity reference coordinate system axis. All major direction clusters are sorted in descending order of their planar area, and the set of walls belonging to a major direction cluster is designated as PlaneSet1. S603: For point cloud areas that do not satisfy condition L1, a normal direction guided Region-Growing region expansion algorithm is executed once to perform complementary reconstruction of the wall surface. For small faces divided by recesses, furniture occlusions, etc., if the angle between the normal of the small face and the adjacent face is less than 5° and the plane distance error with the adjacent face is less than 5 cm, the small face and the adjacent face are joined to obtain a continuous plane. This set of continuous planes is called PlaneSet2. If the area of ​​any plane in PlaneSet2 is 0.3 m, 2 If the above conditions are satisfied and the angle formed by the normal to the plane is equal to or smaller than the set threshold σ with respect to the angle formed by the normal to the main direction cluster in PlaneSet1, the plane is added to the wall surface set PlaneSet1. S604: For all wall point clouds in PlaneSet1, calculate the unit normal vectors n i and center of gravity C i where the subscript i indicates the wall number. S605: The unit vector g in the direction of gravity is set as the vertical direction of the absolute reference, and the inclination angle θ of all walls in PlaneSet1 i where θ i =arccos|n i ·g| and the wall slope rate ε i where ε i =tanθ i ×100%. S606: Wall slope rate ε in PlaneSet1 i If it is 0.7% or more, the wall is marked as a "tilted wall" and the wall tilt flag TiltFlag is set to 1. Otherwise, it is treated as a normal wall and TiltFlag is set to 0. Next, the walls in PlaneSet1 and their tilt flags are saved in the wall list TiltList.

[0012] Furthermore, S7 specifically includes the following processing: S701: The incremental point cloud ΔP acquired by the portable device, the 3D crack list Crack3D_list, and the wall surface list TiltList are packaged and compressed, and transmitted to the mobile terminal.

[0013] S702: The mobile device decompresses the received data, updates and determines the position of the incremental point cloud ΔP in real time according to the viewing angle based on the 3D point cloud coordinate system, and maps it to a world coordinate system for visualization display, thereby maintaining a viewing angle alignment state between the 3D architectural model on the mobile device screen and the real space. The incremental point cloud ΔP is displayed in real time on the mobile device screen, and cracks with DangerFlag=1 are marked as dangerous cracks with red spheres and walls with TiltFlag=1 are marked as tilted walls with yellow triangles based on the danger flag DangerFlag in Crack3D_list and the tilt flag TiltFlag in TiltList. The user can view related information by clicking the red sphere or yellow triangle. Clicking the red sphere displays the actual physical width w of the crack, the temperature difference ΔT, and the time ID. Clicking the yellow triangle displays the wall tilt rate and time ID. Furthermore, the user can enter text notes or take additional on-site photos. In addition, for cracks in the Crack3D_list whose rescan indicator ProjStatus is 1, the mobile device screen will flash "Rescan Crack" and highlight the crack area to be rescanned, prompting the user to rescan. Furthermore, in S602, 2≦n≦4 is set. Furthermore, in S603, the threshold value σ is set to 10°. [Effects of the Invention]

[0014] As can be understood from the above technical means, the beneficial effects of the multi-modal tightly coupled SLAM rapid diagnosis method for brick-concrete mixed structure buildings proposed by the present invention are as follows:

[0015] 1. By simultaneously introducing laser residual, visual residual, thermal image residual, and IMU pre-integration residual into the factor graph, we achieve high-precision real-time localization and map construction even in environments with low light, specular surfaces, or few textures. 2. By integrating multi-modal data from LiDAR, IMU, RGB camera, fisheye camera and infrared thermal camera, and synchronously executing a dual-threshold identification algorithm for crack width and temperature difference as well as wall slope rate calculation, automatic danger detection is realized, effectively reducing the risk of oversight due to a single sensor or single algorithm.

[0016] 3. It performs 3D positioning and visualization in real time for two types of risks, cracks and inclined walls, and supports users in viewing related parameters and rescanning guidance with clicks, greatly improving the efficiency and safety of on-site judgment by inspectors.

[0017] 4. This invention provides a rapid diagnosis method for brick-concrete mixed structure buildings that integrates multi-modal synchronous acquisition, tightly coupled SLAM map construction, real-time hazard detection of cracks and inclined walls, and visualization interaction. It combines real-time performance, accuracy, and ease of use, and can meet the practical needs of rapid safety screening of large-scale dilapidated buildings. [Brief explanation of the drawings]

[0018] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and not limiting of the invention. [Figure 1] This is a flow chart of the rapid diagnosis method for brick-concrete mixed structure buildings using the multi-modal tightly coupled SLAM of the present invention. [Figure 2] 1 is a schematic diagram of the rapid diagnosis method for brick-concrete mixed structure buildings using the multi-modal tightly coupled SLAM of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0019] In order to clarify the purpose, technical configuration, and effects of the embodiments of the present invention, the technical content of the present invention will be described in detail below based on the embodiments with reference to the accompanying drawings. Note that the embodiment described below is one aspect of the present invention and is not intended to limit the present invention. Other embodiments that can be conceived by a person skilled in the art without requiring creative effort based on the disclosure of the present invention are also included in the technical scope of the present invention.

[0020] As shown in Figures 1 and 2, the rapid diagnosis method for brick-concrete mixed structure buildings using multimodal tightly coupled SLAM of the present invention includes the following processes.

[0021] S1: Connect the LiDAR, IMU, RGB camera, fisheye camera, and infrared thermal camera mounted on the mobile device to the same PPS pulse reference signal to unify the time of each sensor. The handheld device (hereinafter referred to as the device) is equipped with five sensors: LiDAR, IMU, RGB camera, fisheye camera, and infrared thermal camera. These are synchronized with the same PPS signal, so that the exposure and scanning of each sensor starts simultaneously. As a result, all sensor data has the same time information (common time identification). S2: The sensor data acquired by the device at each time is converted into a unified device coordinate system based on external and internal parameters, and spatially aligned. Here, the external and internal parameters of each sensor are calibrated in advance, and the same-time data obtained by PPS synchronization is converted from the LiDAR coordinate system, IMU coordinate system, RGB camera coordinate system, fisheye camera coordinate system, and infrared thermal camera coordinate system to a common device coordinate system. That is, each frame acquired by the device includes LiDAR point cloud data, IMU data, RGB image, fisheye image, infrared thermal image, and device attitude, all identified at the same time. S3: The residuals obtained from each sensor (LiDAR residual, visual residual, thermal image residual, IMU pre-integration residual) are written into a factor graph, and high-precision pose and point cloud estimation is performed in real time on the environment map based on SLAM, and the whole is re-optimized during the loop closing process. Specifically, the process is as follows.

[0022] S301: Obtain LiDAR point cloud data in the first keyframe, extract edge and plane feature points using LOAM (i.e., laser odometry and mapping, a typical laser SLAM algorithm), generate a local environment map (a 3D building model in this example), and use the local environment map for feature matching and pose estimation (registration) in subsequent keyframe point clouds. S302: Align the LiDAR point cloud data of the current keyframe to the existing map using the ICP (Iterative Closest Point) or NDT (Normal Distribution Transform) algorithm to estimate the device pose. S303: Extract ORB features from the RGB image and fisheye image in the current keyframe, and extract thermal features based on the temperature gradient from the infrared thermal image. Then, match them with the corresponding features in the previous frame image, calculate the reprojection error, and obtain the visual residual C and thermal image residual H. In this specification, "camera" refers to RGB cameras and fisheye cameras, "image" refers to RGB images, fisheye images and infrared thermal images, and "visible image" refers to RGB images and fisheye images. S304: Using high-frequency IMU data, pre-integrate the acceleration and angular velocity within a short period of time to calculate the inertial increment of the current keyframe relative to the previous keyframe. This inertial increment assists subsequent image feature matching and LiDAR point cloud data registration, improving registration efficiency and convergence stability. In addition, the error term between this inertial increment and the motion difference calculated from the estimated postures of the previous and next keyframes is written into the factor graph as the IMU pre-integration residual I and used for optimization. S305: The laser residual L, visual residual C, thermal image residual H, and IMU pre-integration residual I resulting from the alignment of the LiDAR point cloud data are simultaneously written into a factor graph to construct a nonlinear optimization problem with multi-modal coupling constraints. The Ceres optimizer then performs comprehensive calculations to find the optimal pose P for the current keyframe. t and the incremental point cloud (newly added point cloud data) ΔP is obtained. Next, the posture P t is written into the keyframe sequence Pose_list, and the incremental point cloud ΔP is merged into the existing environment map to achieve real-time update. Here, the key frame sequence Pose_list records the time identification TimeID of the key frame, LiDAR point cloud data, IMU data, RGB camera image, fisheye camera image, infrared thermal image, and device posture. S306: If a feature having high similarity to the current keyframe image is detected in the historical keyframe image, the loop is determined to be closed using a similarity of >0.75 as the criterion. In addition to adding the current keyframe to the factor graph, the poses of all keyframes are readjusted to obtain the corrected pose P new Get P new the original pose P in Pose_list t By replacing it with, we can suppress the accumulated error (drift) and achieve global re-optimization of the environment map.

[0023] S4: Read each key frame, automatically identify cracks, and obtain the center pixel coordinates C(u,v), actual physical width w, temperature difference ΔT, danger flag DangerFlag, and time identification TimeID for each crack, and save them in the crack list Crack_list. Specifically, this includes the following processing: S401: The RGB camera image and infrared thermal camera image for each keyframe are read, the infrared thermal image is reprojected onto the coordinate system of the RGB image, and the temperature information of the infrared thermal image is merged with the pixel position of the RGB image. This generates a four-channel image that adds the temperature channel of the infrared thermal image to the three color channels of R, G, and B. Then, the four-channel image is normalized based on the mean and variance. S402: The four-channel image is input to an improved U-Net network, and a crack distribution image (i.e., crack mask) is finally output. Each pixel value in the crack distribution image is either 0 or 1, where 0 indicates that there is no crack at that pixel position, and 1 indicates that there is a crack. In this example, the improved U-Net network uses the lightweight MobileV2 as the encoder and introduces a multi-scale attention mechanism at the decoder stage. The improved U-Net network is a pre-trained model. S403: The pixel contours of the cracks in the crack distribution image are extracted, and the widest points of all the cracks and their lateral pixel widths are statistically obtained and recorded. Specifically, for each crack, the length of the line segments where the continuous pixel value is 1 in the direction perpendicular to the strike of the crack is counted, and the maximum value of the line segment length is taken as the horizontal pixel width at the widest point of the crack. In other words, the line segment with the maximum length is the widest point of the crack. S404: For all cracks, the horizontal pixel width is converted to the actual physical crack width. When using a portable device, the distance d from the wall surface during imaging is not constant, and in order to ensure the accuracy of the dimension calculation, it is necessary to convert the horizontal pixel width of the widest part of the crack to the actual size based on the pinhole camera model. The camera internal parameters and pixel-corresponding depth information Z obtained in step S2 cam (obtained by projecting the LiDAR point cloud data of the current keyframe) can be used to convert the horizontal pixel width of the widest part of the crack into the actual physical width w of the crack. S405: Average temperature T at the widest part of the crack c and the mean temperature T of the background region b The temperature difference ΔT = T c -T b The background region refers to the area that does not belong to the crack among the 5x5 pixel area around the widest part. S406: If the physical width of the crack is w≧1 mm or the temperature difference ΔT≧1°C, the crack is judged to be a "dangerous crack" and the danger flag DangerFlag is set to 1. In other cases, it is considered to be a normal crack and DangerFlag is set to 0. S407: The central pixel coordinates C(u,v) of the widest part of each crack, the physical width w, the temperature difference ΔT, the danger flag DangerFlag, and the time ID TimeID are stored as one crack record and saved in the crack list Crack_list for use in subsequent 3D projection and position determination.

[0024] S5: Each crack is projected from the image plane onto the environment map to obtain a three-dimensional crack list Crack3D_list, which includes the three-dimensional coordinates (X, Y, Z) of each crack's center point, its physical width w, its temperature difference ΔT, its danger flag DangerFlag, its rescan indicator ProjStatus, and its time identification TimeID.

[0025] The crack list Crack_list generated in step S4 and the device pose P corresponding to the same Time ID in the key frame sequence Pose_list in step S3 t Furthermore, in step S2, each crack is projected from the image coordinate system to the three-dimensional spatial coordinate system of the laser point cloud based on the external and internal parameters of the various sensors. After completing the projection transformation, attribute information such as the three-dimensional coordinates (X, Y, Z) of the center point of each crack, physical width w, temperature difference ΔT, danger flag DangerFlag, rescan indicator ProjStatus, and time identification TimeID is obtained and stored in the three-dimensional crack list Crack3D_list. This information is used to visualize the spatial position in subsequent structural risk analysis.

[0026] S501: Based on the crack list Crack_list generated in step S4, the pose P corresponding to the same Time ID in the key frame sequence Pose_list of step S3 is calculated. t Furthermore, based on the external and internal parameters of each sensor in step S2, the image coordinates of each crack are projected onto the camera coordinate system using a pinhole camera model, and combined with the depth value of the plane obtained by projecting the LiDAR point cloud data of the current key frame, the three-dimensional coordinates (X cam ,Y cam ,Z cam ) is calculated. S502: Coordinates (X cam ,Y cam ,Z cam ) is mapped to the LiDAR coordinate system by coordinate transformation, and the point cloud coordinates of the crack center point (X lid ,Y lid ,Z lid ) so that the crack and laser point cloud can be processed based on a unified geometric reference frame, facilitating subsequent spatial analysis. S503: Point cloud coordinates of the crack center point (X lid ,Y lid ,Z lid ) is mapped to the device coordinate system using rigid body transformation to obtain the position coordinates (X, Y, Z) of the crack center point in three-dimensional space (environment map).The incremental point cloud ΔP is then divided into a voxel map, and each voxel is used as a storage unit for structural features and semantic labels. S504: In the voxel map, the voxel closest to the crack point (crack point in the point cloud) is searched for, and crack attribute information is added to that voxel. The attribute information to be added includes the three-dimensional coordinates (X, Y, Z) of the crack center point, physical width w, temperature difference ΔT, time identification TimeID, danger flag DangerFlag, and rescan indicator ProjStatus (default 0). These attributes are added to the three-dimensional crack list Crack3D_list. S505: In the process of projecting the crack from the image plane to the environment map, the LiDAR point cloud data is used to obtain the effective depth information Z cam If the result is not returned, the rescan indicator for the crack, ProjStatus, is set to 1, and a "Rescan Crack" message is displayed on the mobile device screen to prompt the user to re-photograph the crack area.

[0027] S6: Extract vertical walls close to the building, determine whether the walls are inclined, and generate a wall list TiltList containing wall data and tilt flags TiltFlag. Specifically, the process is as follows: S601: For the incremental point cloud ΔP in each key frame, preprocessing of the point cloud is performed using a voxel filter to remove discrete points and high frequency noise. After that, using the RANSAC plane detection method, approximately vertical walls that satisfy the vertical wall condition L1 are extracted from the point cloud and stored in the approximately vertical wall set PlaneSet0. Here, the condition L1 is that the wall area is 0.5 m 2 This means that the angle α between the wall normal and the direction of gravity is 70°≦α≦110°. S602: The azimuth angles β of all nearly vertical wall surfaces in PlaneSet0 are input into K-Means clustering, and the number of clusters k is set to > 4 to prevent the loss of initial clusters. Then, n main directions (usually the elevation direction, side wall direction, and partition wall direction) are selected based on empirical rules and spatial structure characteristics. Here, the azimuth angle β of a nearly vertical wall surface is the angle between the normal to the wall surface when projected onto a horizontal plane and the coordinate axis reference direction. All main directions are sorted in descending order of planar area, and a wall set for each main direction, PlaneSet1, is obtained. In this embodiment, 2≦n≦4, and the value of n is determined depending on the complexity of the building. S603: For point cloud regions that do not satisfy condition L1, a Region-Growing algorithm based on normal direction is executed once to perform complementary reconstruction of wall surfaces. For small faces divided by door openings, furniture obstructions, etc., if the difference in normal angle between the small face and adjacent face pieces is less than 5° and the difference in distance between the two planes is within 5 cm, these face pieces are integrated into a continuous plane to obtain the complementary plane set PlaneSet2. If the area of ​​any plane in PlaneSet2 is 0.3 m 2 If the above is true and the angle between the normal and the major direction axis in PlaneSet1 is less than a set threshold σ (e.g., 10°), the plane is added to the wall set PlaneSet1, and a complete wall set is constructed for subsequent structural integrity analysis. S604: From all the wall point clouds in PlaneSet1, the unit normal vector n of each wall i and the center of gravity C i (The subscript i is the wall number) is extracted to provide the basic geometric parameters required for subsequent wall inclination rate calculation and structural stability assessment. S605: The unit vector g in the direction of gravity is used as the vertical direction of the absolute reference, and the inclination angle θ of all walls in PlaneSet1 i where θ i =arccos|n i ·g| and the wall slope rate ε i =tanθ i ×100%. S606: Wall slope rate ε in PlaneSet1 i If the difference is 0.7% or more, the wall is determined to be a "tilted wall" and the tilt flag TiltFlag is set to 1. Otherwise, it is considered to be a normal wall and TiltFlag is set to 0. After that, the wall and tilt flag in PlaneSet1 are saved in the wall list TiltList.

[0028] S7: Visualize cracks and sloping walls on mobile devices, display warnings, and enable click-to-check, comment input, and on-site supplementary photography. Specifically, the following is done: S701: The incremental point cloud ΔP, the three-dimensional crack list Crack3D_list, and the wall surface list TiltList acquired by the portable device are packaged and compressed, and sent to the mobile terminal. Here, the point cloud data is spatially hierarchically compressed using an octree to reduce redundant points and improve transmission efficiency. In addition, data such as the 3D crack list Crack3D_list and wall surface list TiltList are encoded using the LZ4 lightweight, high-speed compression method to reduce the overall data volume. S702: The mobile device unpacks the received data and updates and aligns the position and field of view of the incremental point cloud ΔP in real time based on the constructed point cloud coordinate transformation relationship. It accurately overlays the visualization environment, ensuring that the 3D architectural model maintains a consistent spatial relationship with the site. In addition to real-time rendering of the incremental point cloud, the mobile device screen displays dangerous cracks (DangerFlag = 1) as red spheres and tilted walls (TiltFlag = 1) as yellow triangles based on the danger flags DangerFlag in Crack3D_list and the tilt flags TiltFlag in TiltList, emphasizing their spatial location and risk. By tapping the red sphere, users can view the physical width w, temperature difference ΔT, and time ID of the crack. By tapping the yellow triangle, users can view the tilt rate and time ID of the wall, enter comments, and take additional on-site photographs. If ProjStatus in Crack3D_list is 1, the mobile screen flashes "Rescan Crack," prompting them to re-photograph the target crack area with high precision.

[0029] This invention provides a rapid diagnosis method for brick-concrete mixed structure buildings that integrates multi-modal synchronous data acquisition, map construction using tightly coupled SLAM, real-time hazard detection of cracks and sloping walls, and visualization interaction. The invention combines real-time performance, accuracy, and operability, meeting the practical needs of rapid safety inspection of large-scale dilapidated buildings.

[0030] The above description is merely a specific embodiment of the present invention and does not limit the scope of protection of the present invention. Those skilled in the art can easily think of various modifications or replacements within the technical scope of the present invention, all of which are included in the scope of protection of the present invention.

Claims

1. A method for rapid diagnosis of brick and concrete mixed structure buildings using multi-modal tightly coupled SLAM, comprising the steps of: S1: Synchronizing the LiDAR, IMU, RGB camera, fisheye camera, and infrared thermal camera in the portable device to a common time reference using the same PPS signal to complete time synchronization between each sensor; S2: acquiring key frames by a handheld device, reading the external and internal parameters of each sensor, and transforming the multiple sensor data into the device coordinate system to achieve spatial alignment; S3: The laser residuals from the LiDAR, the visual residuals from the RGB camera and the fisheye camera, the thermal image residuals from the infrared thermal camera, and the pre-integration residuals from the IMU in the key frames are written into a factor graph, and high-precision pose and incremental point clouds are estimated and optimized in real time on the environment map based on SLAM, and global calibration is performed during closed-loop detection; S4: A step of reading each key frame, automatically identifying cracks, obtaining the center pixel coordinates C(u, v), physical width w, temperature difference ΔT, danger flag DangerFlag, and time identification TimeID of each crack, and saving them in a crack list Crack_list; S5: A step of projecting each crack from the image plane onto the environment map, obtaining a three-dimensional crack list Crack3D_list, and including the three-dimensional coordinates (X, Y, Z), physical width w, temperature difference ΔT, danger flag DangerFlag, rescan indicator ProjStatus, and time identification TimeID of each crack in the Crack3D_list; S6: A step of extracting substantially vertical wall surfaces of the building, determining whether the wall surfaces are inclined or not, and generating a wall surface list TiltList including wall surface information and a tilt flag TiltFlag; S7: A step of displaying the cracks and sloping wall surfaces on the mobile terminal screen, issuing a warning, and enabling viewing, annotation input, or auxiliary photography by clicking. A rapid diagnosis method for brick and concrete mixed structure buildings using multi-modal tightly coupled SLAM, comprising:

2. 2. The method of claim 1, wherein S3 includes: S301: Obtain LiDAR point cloud data of the first key frame, and extract edge feature points and plane feature points using LOAM (LiDAR Odometry and Mapping) to generate a local environment map. S302: Align the LiDAR point cloud data of the current keyframe with the existing environment map using the Iterative Closest Point (ICP) or Normal Distribution Transform (NDT) algorithm to estimate the device pose. S303: Visual features are extracted from the RGB image and fisheye image of the current keyframe using the ORB algorithm, and feature extraction based on temperature gradient is performed on the infrared thermal image, which is matched with the corresponding features of the previous keyframe image to calculate the reprojection error, and visual residual C and thermal image residual H are obtained. S304: Pre-integration processing is performed on the high-frequency IMU data to calculate the inertial increment between the current key frame and the previous key frame, and the IMU pre-integration residual I is obtained from the difference between the inertial increment and the estimated attitude of the previous and next key frames. S305: The laser residual L, the visual residual C, the thermal image residual H, and the IMU pre-integration residual I are input into the factor graph, and the Ceres optimizer performs nonlinear optimization under multi-modal constraints to obtain the optimal pose P of the current keyframe. t and incremental point cloud ΔP, and the posture P t to the Pose_list, and integrate the incremental point cloud ΔP into the existing environment map.

3. 3. The method of claim 2, wherein step S4 includes: S401: Read the RGB camera image and infrared thermal camera image of each key frame, re-project the infrared image onto the RGB image coordinate system, merge the temperature information of the infrared image at the pixel position of the RGB image, combine the three color channels R, G, B and the infrared thermal temperature channel to generate a four-channel image, and perform normalization processing based on the mean and variance on the four-channel image. S402: The four-channel image is input into an improved U-Net network, and a crack distribution image is output, in which each pixel value in the crack distribution image is 0 or 1, where 0 indicates that there is no crack at that pixel position, and 1 indicates that there is a crack at that pixel position. S403: The pixel contours of the cracks are extracted from the crack distribution image, and the maximum width and horizontal pixel width of each crack are statistically recorded. S404: The horizontal pixel widths of all cracks are converted to actual physical crack widths. S405: Average temperature T of the widest part of all cracks ｃ and the average temperature T of the background area near the widest part ｂ The temperature difference ΔT is obtained as ｃ -T ｂ The background region is the widest 5×5 pixel region, which does not belong to the crack region. S406: If the physical width w of the crack is greater than or equal to 1 mm or the temperature difference ΔT is greater than or equal to 1°C, the crack is determined to be a "dangerous crack" and the danger flag DangerFlag is set to 1; otherwise, the crack is determined to be a normal crack and the DangerFlag is set to 0. S407: Crack records including the central pixel coordinates C(u, v) of the widest part of each crack, the physical width w, the temperature difference ΔT, the danger flag DangerFlag, and the time identification TimeID are packaged and saved in a crack list Crack_list.

4. 4. The method of claim 3, wherein S5 includes: S501: Based on the crack list Crack_list generated in step S4, the pose P corresponding to the same Time ID in the key frame sequence Pose_list in step S3 is calculated. ｔ In step S2, the external and internal parameters of each sensor are read, the pixel coordinates of each crack are back-projected onto the camera coordinate system using a pinhole camera model, and the three-dimensional coordinates (X ｃａｍ , Y ｃａｍ , Z ｃａｍ ) calculating S502: (X ｃａｍ , Y ｃａｍ , Z ｃａｍ ) is projected onto the LiDAR coordinate system by coordinate transformation, and the point cloud coordinates of the crack center point (X ｌｉｄ , Y ｌｉｄ , Z ｌｉｄ ) obtaining the S503: The point group coordinates (X ｌｉｄ , Y ｌｉｄ , Z ｌｉｄ ) into the device coordinate system to obtain the three-dimensional position (X, Y, Z) of the crack center point in the environment map, and divide the incremental point cloud ΔP into a voxel grid, so that each voxel holds a structural feature and a label; S504: A step of searching for a voxel closest to the center point of the crack within the voxel grid, assigning the three-dimensional coordinates (X, Y, Z), physical width w, temperature difference ΔT, time identification TimeID, danger flag DangerFlag, and rescan indicator ProjStatus (default 0) as crack attribute information to the voxel, and adding the attribute information to a three-dimensional crack list Crack3D_list; S505: In the process of projecting the crack from the image plane onto the environment map, effective depth information Z corresponding to the pixel is obtained from the LiDAR point cloud data. ｃａｍ If the crack rescan indicator ProjStatus cannot be obtained, set the crack rescan indicator ProjStatus to 1, and display a message on the mobile terminal indicating that "crack rescan is required" to prompt the user to take additional photos on site.

5. 5. The method of claim 4, wherein step S6 includes: S601: Voxel filtering is performed on the incremental point cloud ΔP of each key frame to remove outliers and high-frequency noise, and then RANSAC plane fitting is performed to satisfy condition L1, i.e., the wall area is 0.5 m ２ and the angle α between the wall normal and the direction of gravity satisfies 70°≦α≦110°, and saving these wall surfaces as a set of approximately vertical wall surfaces PlaneSet0; S602: Clustering the normal direction angles β of all the approximately vertical wall surfaces in PlaneSet0 by K-Means clustering, setting the number of clusters k > 4, selecting n main direction clusters based on empirical rules and spatial structure characteristics, where the normal direction angle β is the angle between the projection vector and the coordinate axis reference direction when the normal vector of the approximately vertical wall surface is projected onto a horizontal plane, and arranging the wall surfaces belonging to each main direction cluster in order of planar area to form a wall surface set PlaneSet1; S603: For the point cloud region that does not satisfy the condition L1, execute the Region-Growing algorithm based on the normal direction once to perform wall surface interpolation reconstruction, and for small planes divided by door openings or furniture occlusions, if the difference in normal angle between the small planes and adjacent planes is less than 5° and the difference in distance between the fitting planes of the small planes and adjacent planes is less than 5 cm, integrate the small planes with the adjacent planes to form a continuous plane, and obtain an interpolation plane set PlaneSet2; Furthermore, if the area of ​​any plane in PlaneSet2 is 0.3 m ２ or more, and the angle difference between the normal of the plane and any normal of the main direction in PlaneSet1 is less than a set threshold σ (e.g., 10°), adding the plane to PlaneSet1; S604: For all wall point groups in PlaneSet1, the unit normal vector n i and the center of gravity C i (where the subscript i indicates the wall number); S605: The unit vector g in the direction of gravity is set as the vertical direction of the absolute reference, and the inclination angle θ of each wall surface in PlaneSet1 is calculated. i (θ i =arccos|n i g|) and further calculate the slope rate ε i (ε i = tan θ i × 100%); S606: For each wall surface in PlaneSet1, the slope rate ε i If the tilt angle is ≧0.7%, the wall surface is determined to be an inclined wall surface and the tilt flag TiltFlag is set to 1, otherwise TiltFlag is set to 0, and the wall surface information and the tilt flag are saved in the wall surface list TiltList.

6. 6. The method of claim 5, wherein S7 includes: S701: Packaging and compressing the incremental point cloud ΔP, the three-dimensional crack list Crack3D_list, and the wall surface list TiltList acquired by the portable device, and transmitting the data to the mobile terminal; S702: The mobile terminal decompresses the received data, and updates and repositions the position of the incremental point cloud ΔP in real time according to the viewing angle based on the established point cloud coordinate transformation relationship, and projects it into the world coordinate system, thereby aligning the 3D architectural model between the mobile terminal screen and the real space; On the mobile terminal screen, in addition to drawing the incremental point cloud in real time, based on the danger flag DangerFlag and the tilt flag TiltFlag in Crack3D_list, displaying cracks with DangerFlag of 1 as dangerous cracks with red spheres and wall surfaces with TiltFlag of 1 as tilted wall surfaces with yellow triangles; The user can view related information by clicking the red sphere or yellow triangle on the screen of the mobile terminal, and can view the physical width w of the crack, the temperature difference ΔT, and the time identification Time ID by clicking the red sphere, and can view the wall slope rate and the time identification Time ID by clicking the yellow triangle, and can also input text notes or take photos on site; Also, if ProjStatus in Crack3D_list is 1, a step of guiding the user to flash "Rescan Crack" on the mobile terminal screen and highlight the crack area that requires supplemental scanning.

7. The method according to claim 5, wherein the number of clusters n in S602 is 2≦n≦4.

8. The method of claim 5, wherein the threshold value in S603 is σ=10°.

Images