A building crack three-dimensional detection method based on double-point cloud sandwich image synchronization

Abstract

The application discloses a kind of based on double-point cloud sandwich image synchronization building crack three-dimensional detection method, belong to three-dimensional measurement technical field, to solve the problem of traditional building crack detection method insufficient accuracy, low efficiency and difficult to obtain crack depth information.The method is by synchronously arranging two point cloud acquisition devices on both sides of crack area, double-view synchronous scanning is carried out to crack area to obtain registration point cloud data, and noise is removed and feature extraction is carried out to point cloud in combination with deep learning algorithm, and three-dimensional reconstruction and accurate positioning of crack are realized based on the spatial constraint relationship of double point cloud.The application can effectively improve the accuracy and efficiency of building crack detection by using the above technical scheme, and obtain the depth and spatial topography information of crack, to provide more comprehensive and accurate data support for building structure safety evaluation.

Description

Technical Field

[0001] This invention belongs to the field of three-dimensional measurement technology, specifically, it relates to a three-dimensional detection method for building cracks based on image synchronization of dual-point cloud clips. Background Technology

[0002] With the continuous advancement of smart city and building information technology (BIT) construction, building structural health monitoring has become one of the core research topics in civil engineering and building operation and maintenance. Building cracks, as an important indicator of structural damage, directly reflect the safety status of buildings through their development and evolution. Therefore, achieving accurate detection and quantitative analysis of cracks is of significant practical importance. In recent years, building inspection technology integrating lidar point cloud and image data has developed rapidly. By leveraging the accuracy advantages of point clouds in spatial measurement and the information advantages of images in texture recognition, it is possible to achieve two-dimensional localization and three-dimensional reconstruction of cracks, providing an effective technical means for building structural health assessment.

[0003] Among these methods, 3D crack detection of buildings based on the fusion of LiDAR and camera technologies has attracted widespread attention in recent years. This type of method integrates a LiDAR sensor and an industrial camera on a detection platform, using LiDAR point clouds to acquire the spatial 3D coordinates of the crack area. Simultaneously, it leverages image data to achieve semantic segmentation and feature extraction of the crack, ultimately achieving precise quantification of crack parameters within a unified spatial coordinate system. However, in practical engineering applications, the time synchronization problem between sensors has become a key bottleneck restricting detection accuracy and reliability. Since LiDAR and cameras are heterogeneous sensors, their operating frequencies, clock references, and data acquisition modes differ significantly. Achieving high-precision time alignment between point cloud data and image data directly determines the accuracy of subsequent cross-modal fusion and 3D reconstruction.

[0004] In existing technologies, time synchronization between LiDAR and cameras mainly relies on hardware synchronization modules or dynamic fusion algorithms based on motion models. Hardware synchronization requires additional dedicated time synchronizers and clock signal generators, significantly increasing system hardware costs and adding complexity to sensor calibration and wiring connections. This makes it poorly suited to the lightweight and rapid detection needs of building sites. Dynamic fusion algorithms typically align time sequences based on motion assumptions between consecutive frames. These algorithms are primarily designed for medium-to-high-speed motion scenarios such as robot navigation and autonomous driving. Their motion models and pre-integration strategies struggle to achieve accurate synchronization in building crack detection scenarios where sensors are stationary or moving at low speeds. Furthermore, the synchronization error may even increase sharply due to the failure of the motion assumptions. In addition, existing 3D reconstruction methods that fuse point clouds and images lack specific optimization strategies for static acquisition scenarios when dealing with surface defects with elongated geometric features, such as building cracks. The sparsity and noise characteristics of single-frame point clouds are insufficient to meet the accuracy requirements for crack depth measurement. In summary, how to achieve high-precision time synchronization between LiDAR and camera in building crack detection scenarios under low hardware cost conditions, and on this basis, improve the quantification accuracy of crack three-dimensional parameters through static acquisition optimization strategies, has become a technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0005] The purpose of this invention is to provide a three-dimensional detection method for building cracks based on image synchronization using dual-point cloud clips, which mainly solves the problems of insufficient accuracy, low efficiency, and difficulty in obtaining crack depth information in traditional building crack detection methods.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for 3D detection of building cracks based on image synchronization using dual point cloud clips includes the following steps:

[0008] S1, establish a time synchronization mechanism, acquire the first laser point cloud and record the first timestamp T1, then trigger the camera to acquire the crack image and record the image timestamp Timg, then acquire the second laser point cloud and record the second timestamp T2;

[0009] S2, calculate the intermediate timestamp Tmid of the two point clouds based on T1 and T2 and calibrate the image timestamp to Tmid. Construct a three-dimensional point cloud map based on the point cloud corresponding to the calibrated Tmid, and establish the mapping relationship between image pixels and three-dimensional point cloud by combining the pre-calibrated radar-camera extrinsic parameters.

[0010] S3, the first laser point cloud and the second laser point cloud are superimposed and denoised to improve the point cloud density in the crack area;

[0011] S4. The crack image is segmented using the trained crack segmentation model to obtain the crack mask. The crack mask region is then spatially searched and matched with the 3D point cloud to establish an accurate correspondence.

[0012] S5 calculates the length, width, and depth parameters of the crack and generates a crack detection report. For scenarios where the sensor moving speed is less than or equal to a set threshold, the motion offset between T1 and T2 is compensated using the pre-integrated data from the inertial measurement unit before calculating Tmid.

[0013] Furthermore, in step S1, the specific process of establishing the time synchronization mechanism is as follows:

[0014] The system controls the lidar to acquire the first frame of point cloud data and records T1 using a hardware timer;

[0015] After completing the acquisition of the first frame of point cloud, the system keeps the sensor stationary and then sends a shooting trigger signal to the camera. After receiving the trigger signal, the camera completes image acquisition within the preset exposure delay time.

[0016] After completing the crack image acquisition, the system controls the lidar again to acquire the second frame of point cloud data and records T2;

[0017] The system calculates Tmid based on T1 and T2, and uses this Tmid to calibrate the crack image timestamp from the original Timg to Tmid in order to eliminate the drift error between the camera clock and the lidar clock.

[0018] Furthermore, in step S2, the construction process of the 3D point cloud map is as follows:

[0019] The pose estimation is achieved by tightly coupling and fusing laser point cloud data with inertial measurement unit data using the lidar inertial odometry method.

[0020] An incremental map management strategy is adopted to gradually add the scanned and registered point cloud data to the global map;

[0021] The original point cloud is downsampled using a voxel grid filtering method to reduce the amount of data and computational complexity;

[0022] An improved version of the iterative nearest point algorithm is used to match the point cloud of the current scan frame with the point cloud in the local map to estimate the pose transformation;

[0023] The ray casting method is used to find the 3D points in the point cloud map that correspond to the image pixels.

[0024] Furthermore, in this invention, the process of establishing the mapping relationship between the image pixels and the three-dimensional point cloud is as follows:

[0025] Normalized coordinates are obtained by projecting the pixel coordinates of the crack image onto the normalized plane using the camera intrinsic parameter matrix.

[0026] The direction vector of the three-dimensional ray is obtained by transforming the normalized coordinates to the lidar coordinate system using the rotation matrix of the radar-camera extrinsic parameters; the radar-camera extrinsic parameters are pre-calibrated using a joint optimization method with a calibration board.

[0027] Ray projection is performed on the point cloud map along the ray direction, and the three-dimensional point corresponding to the image pixel depth is quickly located by using a binary search method.

[0028] Further, in step S3, the superposition denoising processing method is as follows:

[0029] The point cloud registration algorithm is used to transform the second frame point cloud into the coordinate system of the first frame point cloud to achieve spatial alignment;

[0030] Two frames of point clouds are merged into a higher-density composite point cloud using a point cloud fusion algorithm.

[0031] Statistical filtering is used to identify and remove noise points. The Euclidean distance between each point and all points in its neighborhood is calculated. If the mean Euclidean distance between a point and its neighbors is greater than a set threshold, the point is identified as a noise point and removed.

[0032] A pass-through filter is used to remove outliers in a certain direction from the point cloud;

[0033] Conditional filtering is used to filter and remove points with excessively low reflection intensity or abnormal depth values ​​based on the attribute characteristics of the points.

[0034] Further, in step S4, the crack segmentation model adopts a fully convolutional network architecture, which is an encoder-decoder structure. The encoder part uses a residual network as the backbone feature extraction network, which contains multiple residual stages. Each stage contains multiple residual units. The skip connection design of the residual units is used to alleviate the gradient vanishing problem of deep networks. The feature map resolution of each stage of the encoder decreases step by step, and the number of channels increases step by step. The decoder part gradually restores the spatial resolution of the feature map through transposed convolution operation, and fuses the feature maps of the corresponding layers of the encoder through skip connections. The decoder outputs a crack segmentation mask with the same resolution as the input image. The mask is a binary image, where a pixel value of one represents the crack region and a pixel value of zero represents the background region.

[0035] Furthermore, in step S5, the calculation method for the length, width, and depth parameters of the crack is as follows:

[0036] Extract the coordinates of all three-dimensional points in the crack mask projection region to form a three-dimensional point set, calculate the Euclidean distance between each pair of points in the point set and take the maximum value as the crack length;

[0037] Multiple sampling points are obtained by sampling the center line of the crack at equal intervals. The depth difference between the two edge points of the crack on both sides in the normal direction of each sampling point is extracted, and the mean of the depth difference of all sampling points is calculated as the average width of the crack.

[0038] The crack surface points are fitted to a plane equation using the least squares method. The distances from all crack surface points to this plane are calculated, and the average value is taken as the crack depth.

[0039] Furthermore, in step S5, the specific method for motion offset compensation is as follows:

[0040] The rotation increment and displacement increment are obtained by integrating the raw angular velocity and acceleration data of the inertial measurement unit during the time period from T1 to T2.

[0041] Motion compensation is achieved by applying rotation and translation transformations to each 3D point in the second point cloud using rotation and displacement increments.

[0042] After completing motion compensation, calculate the calibrated intermediate timestamp according to the correction formula;

[0043] The pre-integration results of the inertial measurement unit are optimized and corrected online by combining visual odometry. A sliding window filter is used to fuse the pose estimation of visual odometry with the pre-integration results of the inertial measurement unit.

[0044] Furthermore, in this invention, the online optimization and correction process is as follows:

[0045] A continuous-time trajectory optimization method is used to globally adjust the historical poses and construct a pose graph optimization problem, where nodes are the sensor poses at each time step, and edges are the pre-integration constraints of the inertial measurement unit and the visual odometry constraints.

[0046] High-precision sensor trajectories are obtained by solving the global optimal solution of the pose graph using a nonlinear optimization algorithm.

[0047] Based on the optimized trajectory, distortion correction is performed on the point cloud of each frame to eliminate the point cloud deformation caused by motion. The distortion correction process uses a linear interpolation method to interpolate the pose according to the timestamp of each point and transform the point cloud coordinates to a unified reference time point.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] (1) This invention establishes a time synchronization mechanism for dual point cloud images, and simultaneously arranges two point cloud acquisition devices on both sides of the crack area. The midpoint of the dual point cloud acquisition time is used as the intermediate timestamp to calibrate the image timestamp. No additional hardware synchronization module or dedicated time synchronizer is required. The drift error between the camera clock and the lidar clock can be eliminated by software algorithm alone. While significantly reducing the system hardware cost and deployment complexity, it achieves high-precision time alignment between point cloud and image data, providing a reliable technical guarantee for lightweight and rapid detection on construction sites.

[0050] (2) This invention performs noise reduction processing on the first laser point cloud and the second laser point cloud by superimposing them. It utilizes the spatial complementarity of the two point clouds to fuse them into a composite point cloud with higher density. It also combines statistical filtering, direct filtering and conditional filtering to remove noise points, effectively making up for the measurement defects caused by the sparsity of the single-frame point cloud. This makes the three-dimensional geometric features of the crack surface more complete and clear, significantly improving the quantitative accuracy of three-dimensional parameters such as crack depth, and providing more reliable data support for the accurate location of cracks and structural safety assessment.

[0051] (3) This invention is designed for building crack detection scenarios where the sensor is stationary or moving at low speed. It uses the pre-integrated data of the inertial measurement unit to compensate for the motion offset within the time window of the dual point cloud acquisition before calculating the intermediate timestamp. It also uses the sliding window fusion of visual odometry and pre-integration of the inertial measurement unit and continuous time trajectory optimization to correct and distort the pose. This avoids the problem of increased synchronization error caused by the failure of motion assumptions in traditional dynamic fusion algorithms. It makes the detection method applicable to both fixed-point detection and mobile inspection on the building site, expanding the scope of application and enhancing the practicality of engineering.

[0052] (4) The present invention uses the crack mask obtained by segmenting the crack image by the trained crack segmentation model and performs spatial search matching with the three-dimensional point cloud after superposition and denoising to establish a precise correspondence between image pixels and three-dimensional point cloud. Then, the length, width and depth parameters of the crack are quantified by Euclidean distance calculation, normal depth difference statistics and plane fitting respectively, realizing multi-dimensional and high-precision quantitative characterization of crack three-dimensional morphology, and providing a comprehensive evaluation basis for building structure health monitoring covering multi-dimensional indicators such as length, width and depth. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the overall process of the method of the present invention.

[0054] Figure 2 This is a schematic diagram of the time synchronization mechanism in the method of the present invention.

[0055] Figure 3 This is a schematic diagram illustrating the construction process of a three-dimensional point cloud map in the method of the present invention.

[0056] Figure 4 This is a schematic diagram illustrating the process of establishing the mapping relationship between image pixels and three-dimensional point clouds in the method of this invention.

[0057] Figure 5 This is a schematic diagram illustrating the specific method flow for motion offset compensation in this invention.

[0058] Figure 6 This is a schematic diagram of the online optimization and correction process in this invention.

[0059] Figure 7 This is a diagram illustrating the effect of crack image segmentation masking in this invention. Detailed Implementation

[0060] The present invention will be further described below with reference to the accompanying drawings and embodiments. The embodiments of the present invention include, but are not limited to, the following embodiments.

[0061] like Figure 1 As shown, this invention discloses a three-dimensional detection method for building cracks based on dual-point cloud image synchronization. The method first establishes a time synchronization mechanism. With the sensor stationary, the system controls the lidar to first acquire the first frame of point cloud data and record the timestamp T1 of that frame. This timestamp is acquired through a hardware timer built into the lidar. The timer's reference clock frequency is typically 100MHz or higher to ensure nanosecond-level time resolution. The acquisition range of the first frame of point cloud is determined by the type of lidar. If a Livox Mid-360 lidar is used, its horizontal field of view is 360°, its vertical field of view is 59°, and its maximum effective ranging distance is 100m. The time required to acquire a single frame of point cloud is approximately 0.1s. During the acquisition process, the lidar sends the raw point cloud data to the onboard computing unit via the UDP network protocol. Each frame of point cloud data contains three-dimensional coordinate information (X, Y, Z) and reflection intensity information.

[0062] After acquiring the first frame of point cloud data, the system keeps the sensor stationary and then sends a trigger signal to the camera. Upon receiving the trigger signal, the camera completes image acquisition within a preset exposure delay time. This exposure delay time is typically set between 1ms and 5ms, depending on the camera type and parameter settings. The crack image captured by the camera has a resolution of 1920×1080 pixels or higher, and the image format is RGB color or grayscale. The camera transmits the image data to the computing unit via the GigE Vision interface. The image's original timestamp, Timg, is recorded by a precise clock inside the camera, which has an inherent clock drift error compared to the LiDAR's hardware clock.

[0063] After acquiring the crack image, the system controls the lidar to acquire a second frame of point cloud data and records the timestamp T2 of this frame. The acquisition parameters of the second frame of point cloud are consistent with those of the first frame to ensure comparability in spatial coverage and data density. The lidar acquisition time for the second frame of point cloud is the same as that for the first frame, approximately 0.1 seconds. The time interval between the two frames is ΔT = T2 - T1, which is determined based on the lidar's scanning frequency. When the scanning frequency is 10Hz, ΔT is approximately 0.1 seconds.

[0064] The system calculates the intermediate timestamp Tmid of the two laser point clouds based on the timestamp T1 of the first laser point cloud and the timestamp T2 of the second laser point cloud. The formula for calculating Tmid is:

[0065]

[0066] The Tmid calculated using this formula represents the midpoint between the acquisition times of two point cloud frames. Subsequently, the system calibrates the timestamp of the crack image from the original Timg to Tmid, thereby eliminating the drift error between the camera clock and the LiDAR clock. The calibrated timestamp Tmid accurately corresponds to the geometric center position of the dual point cloud time window, ensuring optimal synchronization between the crack image and the point cloud data in the time dimension. This time synchronization scheme requires no additional hardware synchronization module or dedicated time synchronizer; it achieves timestamp calibration solely through software algorithms, keeping the time synchronization error within 10ms.

[0067] like Figure 2 As shown, after the lidar completes the acquisition of the first frame of point cloud data at time T1, the system immediately sends a synchronization trigger signal to the camera. Upon receiving the trigger signal, the camera performs exposure and capture according to the preset exposure delay time. This exposure delay time, along with the total duration of the inertial measurement unit's pre-integration period, is controlled within 10ms, ensuring that the camera exposure occurs near the midpoint between T1 and T2. This synchronization trigger control mechanism guarantees a strict consistency in the time sequence of the preceding point cloud acquisition, image capture, and subsequent point cloud acquisition, providing a reliable time reference for subsequent cross-modal data fusion.

[0068] The system constructs a 3D point cloud map based on the Fast-LIO2 (Fast-LIO2) method. Fast-LIO2 is a highly efficient lidar inertial odometry method. Its core idea is to tightly couple and fuse lidar point cloud data with inertial measurement unit (IMU) data to achieve high-frequency, high-precision pose estimation. The algorithm employs an incremental map management strategy, gradually adding the scanned and registered point cloud data to the global map to construct a dense 3D point cloud map.

[0069] In the specific implementation process, the system first initializes the extrinsic parameters of the lidar and the inertial measurement unit. These extrinsic parameters include the translation vector and rotation quaternion of the lidar relative to the inertial measurement unit, which are obtained through pre-calibration experiments. Upon system startup, the system reads the raw data (angular velocity and acceleration) from the inertial measurement unit and calculates the relative motion increment between adjacent time points using a pre-integration algorithm. Simultaneously, the system receives real-time point cloud data from the lidar and downsamples the raw point cloud using a voxel grid filtering method to reduce data volume and computational complexity.

[0070] The scanning registration process employs an improved version of the Iterative Nearest Point (INP) algorithm. The system matches the point cloud of the current scan frame with the point cloud in the local map, estimating the pose transformation of the current scan frame relative to the local map by minimizing the distance error between the point clouds. The INP algorithm typically iterates 20 times, with a convergence threshold of 0.01m, ensuring sub-centimeter level registration accuracy. During each iteration, the system uses a KD-Tree data structure to accelerate the nearest neighbor search, with a search radius set to 0.5m.

[0071] After a new scan frame is aligned with an existing map, the system adds the registered point cloud data to the global map. The global map employs an incremental update strategy, adding only point clouds from the current scan frame that do not overlap with existing maps to avoid redundant data storage. The map's storage structure uses an octree or voxel hash table for efficient spatial querying and storage management. The point cloud density of the global map is determined based on the LiDAR's scanning resolution and the scene's complexity; in close-range areas (0 to 5m), the point cloud density is approximately 5000 points per cubic meter, while in distant-range areas (5 to 10m), the point cloud density is approximately 1000 points per cubic meter.

[0072] Using the calibrated Tmid corresponding to the spatial coordinates in the point cloud map, a mapping relationship between image pixels and the 3D point cloud is established in conjunction with pre-calibrated radar-camera extrinsic parameters. The radar-camera extrinsic parameters include the translation vector and rotation matrix of the camera relative to the lidar, which are pre-calibrated using a joint optimization method with a calibration board. The calibration process uses a checkerboard calibration board or an Apriltag calibration target, acquiring lidar point cloud and image data from the calibration board at different poses, and solving for the extrinsic parameters using a joint optimization algorithm. The translation error of the calibration results is less than 2 mm, and the rotation error is less than 0.1°, ensuring that the accuracy of the pixel-to-point cloud mapping meets the requirements for crack detection.

[0073] The specific implementation process of cross-modal data fusion is as follows: First, the pixel coordinates of the crack image are projected onto a normalized plane through the camera intrinsic parameter matrix to obtain normalized coordinates. Then, the normalized coordinates are transformed into the lidar coordinate system through the rotation matrix of the radar-camera extrinsic parameters to obtain the direction vector of the three-dimensional ray. Finally, ray projection is performed along the ray direction in the point cloud map to find the three-dimensional point corresponding to the image pixel. Specifically, the search range of the ray projection is set to 0.1m to 20m, the search step size is set to 0.01m, and the three-dimensional point corresponding to the depth of the image pixel is quickly located using a binary search method.

[0074] Subsequently, the system performs superposition denoising on the first and second laser point clouds to improve the measurement accuracy of the point cloud in the crack region. The core idea of ​​superposition denoising is to utilize the spatial complementarity of the two point clouds and merge them into a higher-density composite point cloud through a point cloud fusion algorithm, while eliminating noise introduced by the environment and sensors.

[0075] The point cloud overlay process begins with spatial alignment of the two point cloud frames. Since the two frames were acquired at different times, the sensors may exhibit slight pose variations. Therefore, before overlaying, a point cloud registration algorithm is needed to transform the second frame's point cloud into the coordinate system of the first frame. This registration algorithm employs a variant of the iterative nearest-point algorithm, estimating rigid body transformation parameters by minimizing the distance error between corresponding points in the two frames. The transformation parameters include three-axis rotation angles and three-axis translational displacements, and the optimal transformation matrix is ​​solved using singular value decomposition.

[0076] The superimposed point cloud is then denoised using statistical filtering. The core idea of ​​statistical filtering is to assume that the distances between the neighboring points of each point in the point cloud follow a Gaussian distribution, and to identify and remove noise points based on statistical characteristics. Specifically, for each point in the superimposed point cloud, the Euclidean distances between it and all its neighboring points are calculated, and the mean and standard deviation of these distances are calculated. If the mean distance between a point and its neighboring points is greater than a set threshold (threshold = mean + k × standard deviation, where k is a multiple of the standard deviation, set to 2.0), then the point is identified as a noise point and removed from the point cloud. The number of neighboring points for statistical filtering is set to 20; this parameter selection considers a balance between computational efficiency and denoising effectiveness.

[0077] Denoising processing also includes pass-through filtering and conditional filtering. Pass-through filtering is used to remove outliers in a point cloud in a certain direction. For example, if the vertical height range is set to 0 to 10m, floating points above or below this range are removed. Conditional filtering is used to filter points based on their attribute characteristics, such as removing points with excessively low reflection intensity or abnormal depth values.

[0078] After superposition denoising, the point cloud density in the crack region was significantly improved. In the original single-frame point cloud, the point cloud density on the crack surface was approximately 50 points per square decimeter. After superposition of two point clouds, the density increased to over 100 points per square decimeter. This increased point cloud density makes the three-dimensional geometric features of the crack surface more complete and clear, providing a more reliable data foundation for subsequent parameter quantization. Experiments show that superposition denoising improves the depth measurement accuracy of the crack point cloud by 15% to 20%, reducing the depth measurement error from ±5mm to within ±4mm.

[0079] The system uses a pre-trained crack segmentation model to segment crack images and obtain crack masks. This crack segmentation model employs a fully convolutional network architecture, specifically an encoder-decoder structure. The encoder part uses a residual network as the backbone feature extraction network, containing four residual stages, each with multiple residual units. The skip connection design of the residual units effectively alleviates the gradient vanishing problem in deep networks, enabling the network to learn deeper image features. The feature map resolution of each stage of the encoder is progressively reduced to 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the input image, while the number of channels progressively increases to 64, 128, 256, and 512.

[0080] The decoder gradually restores the spatial resolution of the feature maps through transposed convolution operations, and simultaneously fuses the feature maps from corresponding encoder layers through skip connections. The feature map resolution at each stage of the decoder is progressively restored to 1 / 8, 1 / 4, and 1 / 2 of the input image resolution. Finally, a 1×1 convolutional layer outputs a crack segmentation mask with the same resolution as the input image. The decoder output mask is a binary image, where a pixel value of 1 represents the crack region and a pixel value of 0 represents the background region.

[0081] The crack segmentation model was trained on a large-scale architectural crack image dataset. This dataset covers various building material surfaces, including concrete, brick walls, and plaster layers, with over 50,000 images. The image resolution was uniformly adjusted to 512×512 pixels for batch processing. Data augmentation was performed on the dataset for different lighting conditions, occlusion levels, and crack morphologies, including random brightness adjustment (range -30% to +30%), random contrast adjustment (range 0.8 to 1.2), random cropping and rotation (angle range -15 degrees to +15 degrees), and random occlusion simulation (occlusion ratio range 5% to 20%). The model training employed a joint optimization strategy using cross-entropy loss and dice loss functions to address class imbalance. During training, the initial learning rate was set to 0.001, a cosine annealing learning rate scheduling strategy was used, the batch size was set to 16, and the total training epochs were 100. After training, the model has a crack mask extraction pixel accuracy of over 95% on the test set, can identify cracks with a width of 0.1 mm or more, has a recall rate of over 90%, and a false positive rate of less than 5%.

[0082] The crack mask pixel coordinates are converted into 3D ray direction vectors using the camera intrinsic and extrinsic parameters. The camera intrinsic matrix includes the focal length. , and the main point , Four parameters, obtained in advance through camera calibration. Pixel coordinates ( , ) to normalized plane coordinates ( The conversion formula for ) is:

[0083]

[0084] Normalized planar coordinates ( The 3D ray direction vector dir is obtained by transforming the radar-camera extrinsic parameters into the lidar coordinate system using the rotation matrix R and translation vector t. This direction vector satisfies... .

[0085] Spatial search and matching are performed on the denoised point cloud map to establish a precise correspondence between the crack mask region and the 3D point cloud. The spatial search employs a ray casting method, starting from the camera's optical center and proceeding step-by-step along the 3D ray direction. The search starting point is the camera's optical center, the search direction is the normalized result of the ray direction vector, the search endpoint is a preset maximum depth value (usually 20m), and the search step size is set to 0.005m to ensure matching accuracy. At each search step, the distance from the current search point to its nearest neighbor in the point cloud map is calculated. If the distance is less than a set threshold (set to 0.02m), the nearest neighbor is marked as the 3D point corresponding to the current pixel. By traversing all pixel coordinates in the crack mask, the system can establish a complete mapping relationship between the crack image region and the 3D point cloud.

[0086] The quantitative parameters for crack calculation include three dimensions: length, width, and depth. Crack length is defined as the Euclidean distance between the two farthest 3D points in the crack mask projection area. The specific calculation process is as follows: First, extract the coordinates of all 3D points in the crack mask projection area to form a 3D point set P; then calculate the Euclidean distance between each pair of points in point set P, and take the maximum value after traversing all distances as the crack length. Crack width is defined as the average normal distance of multiple sampling points on the crack cross-section. The specific method for cross-section sampling is as follows: sample the crack centerline at equal intervals (interval set to 2mm) to obtain N sampling points; for each sampling point, extract the depth difference between the two edge points on either side of the crack in the normal direction; calculate the mean of the depth differences of all sampling points as the average crack width. Crack depth is defined as the average perpendicular distance from each point on the crack surface to the plane after fitting the plane using the least squares method. The equation of the fitted plane is ax + by + cz + d = 0. The plane parameters a, b, c, and d are solved by minimizing the sum of the squares of the distances from each point to the plane; then, the distances from all crack surface points to the plane are calculated, and the average value is taken as the crack depth.

[0087] The system ultimately generates a crack detection report containing quantitative parameters such as crack length, width, and depth, along with the corresponding image region. The report uses the JSON standard data format and includes a unique crack identifier, 3D coordinates, quantitative parameter values, confidence score, and a corresponding crack image screenshot. The report can be exported to PDF or CSV format for easy reference by engineers for subsequent crack assessment and archiving.

[0088] like Figure 5 , Figure 6 As shown, when the sensor moves at a speed of less than or equal to 0.5 m / s, the system compensates for the motion offset between T1 and T2 using pre-integrated data from the inertial measurement unit, and then calculates Tmid to achieve synchronous calibration in dynamic scenarios. This extended scheme ensures that the detection method is applicable to both fixed-point detection and mobile inspection scenarios on construction sites.

[0089] The core idea of ​​inertial measurement unit (IMU) pre-integration is to calculate the cumulative motion of the IMU during the time interval T1 to T2. The pre-integration algorithm integrates the original angular velocity and acceleration data to obtain the rotation increment during that time interval. and displacement increment Specifically, angular velocity data Integrating along the time axis, the integration formula is:

[0090]

[0091] The displacement increment is obtained by a second integral of the acceleration data, but the effect of gravitational acceleration must first be subtracted. The integral formula is:

[0092]

[0093] in The original acceleration data is measured by the accelerometer, and g is the gravitational acceleration vector.

[0094] Using rotational increments and displacement increment Motion compensation is performed on the spatial coordinates of the second point cloud. The specific method of motion compensation is as follows: For each 3D point P2 in the second point cloud, a rotation transformation and a translation transformation are applied to map it to the coordinate system of the first point cloud. The transformed points... satisfy:

[0095]

[0096] Rotation transformations are achieved using a rotation matrix, which consists of rotation increments. The quaternion representation is derived from this. Translation transformation is achieved through displacement increments. Direct application. After motion compensation, the second point cloud and the first point cloud are spatially aligned, and the two point clouds can be regarded as virtual data collected at the same time.

[0097] After motion compensation, the system calculates the calibrated image timestamp according to Tmid = (T1 + T2) / 2. Since there is a motion offset between T1 and T2, directly using the original T1 and T2 to calculate Tmid will lead to time calibration errors. After motion compensation of the second point cloud using pre-integrated data from the inertial measurement unit, the compensated point cloud timestamp can be considered as... ,in This is the time correction amount corresponding to motion compensation. Therefore, the calibrated intermediate timestamp satisfy:

[0098]

[0099] In scenarios where the sensor moving speed is less than or equal to 0.5 m / s, this motion compensation method can effectively eliminate the impact of motion offset on time synchronization and control the time synchronization error within 15 ms.

[0100] The system also incorporates visual odometry to perform online optimization and correction of the inertial measurement unit's (IMU) pre-integration results. Visual odometry estimates the relative motion of the sensors by analyzing the feature point matching relationships between consecutive image sequences. The pose estimation from visual odometry is fused with the IMU pre-integration results using a sliding window filter. The sliding window size is set to 10 frames, and the filter state includes the sensor's pose, velocity, and IMU bias. The fused motion estimation accuracy is superior to either single IMU pre-integration or visual odometry, effectively reducing the cumulative drift of motion compensation.

[0101] In mobile inspection mode, the system also employs a continuous-time trajectory optimization method to globally adjust historical poses. Specifically, the system maintains a pose graph optimization problem, where nodes represent sensor poses at various time points, and edges represent inertial measurement unit pre-integration constraints and visual odometry constraints. A nonlinear optimization algorithm (such as g2o or GTSAM) is used to solve for the global optimal solution of the pose graph, obtaining a high-precision sensor trajectory. Based on the optimized trajectory, distortion correction is performed on the point cloud of each frame to eliminate point cloud deformation caused by motion. The distortion correction uses a linear interpolation method, interpolating the pose based on the timestamp of each point to transform the point cloud coordinates to a unified reference time point.

[0102] Taking the crack detection of a concrete column in an industrial plant as an example, the inspectors fixed the sensor module, consisting of a LiDAR and a camera, on a tripod to ensure that the sensor remained stationary during the detection process. The LiDAR used was a Livox Mid-360 device, and the camera was a 12-megapixel industrial camera with an 8mm lens focal length.

[0103] After the detection begins, the system first executes the time synchronization mechanism. The lidar acquires the first frame of point cloud data, with a timestamp T1 = 1000.0 ms. The system keeps the sensor stationary and controls the camera to capture an image of the cracks in the concrete column; the original image timestamp Timg = 1001.3 ms. The camera exposure delay is 2 ms. The lidar then acquires the second frame of point cloud data, with a timestamp T2 = 1100.0 ms. The system calculates Tmid = (1000.0 + 1100.0) / 2 = 1050.0 ms and calibrates the image timestamp to 1050.0 ms. After calibration, the image timestamp is aligned with the center of the point cloud time window, and the time synchronization error is 50 ms, meeting the synchronization accuracy requirements (guaranteed by the synchronization trigger mechanism).

[0104] Subsequently, as Figure 3As shown, the system constructs a 3D point cloud map based on the Fast-LIO2 algorithm. The spatial resolution of the point cloud map is 0.05m, and the voxel grid size is 0.1m. The system uses the calibrated Tmid = 1050.0ms corresponding to the spatial coordinates in the point cloud map, combined with pre-calibrated radar-camera extrinsic parameters, to establish a mapping relationship between pixels and the point cloud, as shown below. Figure 4 As shown. The extrinsic parameter calibration results show that the translation vector of the camera relative to the lidar is [-0.05m, 0.02m, 0.01m], and the rotation matrix is ​​expressed by Euler angles as [0.02rad, -0.01rad, 0.005rad].

[0105] Subsequently, the system performed a superposition and denoising process on the two point cloud frames. After superposition, the total number of point clouds was approximately 80,000 points. After statistical filtering denoising, the number of point clouds was reduced to approximately 75,000 points. The denoising process identified and removed approximately 5,000 noise points. After denoising, the point cloud density in the crack region increased from 55 points per square decimeter to 110 points per square decimeter.

[0106] Then, the system processes the crack image using a crack segmentation model. For example... Figure 7 As shown, the model input image size is 512×512 pixels, and the output crack mask resolution is 512×512 pixels. The model identifies a crack region accounting for approximately 3.5% of the pixels, corresponding to a crack with a width of approximately 2mm in the image. The crack mask pixel coordinates are converted into 3D ray directions using camera intrinsic and extrinsic parameters, and a matching search is performed on the point cloud map. The nearest neighbor distance threshold for the search matching is 0.02m, and a total of 156 points were successfully matched. The quantitative parameters of the crack are calculated as follows: length 0.42m, average width 2.3mm, and depth 1.8mm. The system generates a detection report, including the crack's unique identifier, detection time, 3D coordinates (X=1.25m, Y=0.85m, Z=1.62m), quantitative parameter values, and a screenshot of the crack image.

[0107] If the inspection personnel need to move around for inspection, the low-speed scenario extension plan is implemented. The sensor moves along the concrete wall at a speed of 0.3 m / s, and the inertial measurement unit collects angular velocity and acceleration data during the time period T1 to T2, with a sampling frequency of 200 Hz. The rotation increment is calculated through pre-integration. =[0.01rad, 0.005rad, 0.002rad], displacement increment =[0.015m, 0.008m, 0.001m]. After motion compensation of the second point cloud, the system calculates the calibrated intermediate timestamp Tmid'=1047.5ms, with the time synchronization error controlled within 12ms. The visual odometry performs online optimization of the pre-integration results, improving the motion estimation accuracy by approximately 20% and reducing the cumulative drift by approximately 30% after fusion.

[0108] Through the above design, this invention uses the midpoint of the dual point cloud acquisition time as the intermediate timestamp to calibrate the image timestamp. There is no need to configure additional hardware synchronization modules or dedicated time synchronizers. The drift error between the camera clock and the lidar clock can be eliminated by software algorithms alone. While significantly reducing the system hardware cost and deployment complexity, it achieves high-precision time alignment between point cloud and image data, providing a reliable technical guarantee for lightweight and rapid detection on construction sites.

[0109] The above embodiments are merely one of the preferred embodiments of the present invention and should not be used to limit the scope of protection of the present invention. Any modifications or refinements made to the main design concept and spirit of the present invention that are not of substantial significance, but solve the same technical problem as the present invention, should be included within the scope of protection of the present invention.

Claims

1. A method for three-dimensional detection of building cracks based on image synchronization using dual point cloud clips, characterized in that, Includes the following steps: S1, establish a time synchronization mechanism, acquire the first laser point cloud and record the first timestamp T1, then trigger the camera to acquire the crack image and record the image timestamp Timg, then acquire the second laser point cloud and record the second timestamp T2; S2, calculate the intermediate timestamp Tmid of the two point clouds based on T1 and T2 and calibrate the image timestamp to Tmid. Construct a three-dimensional point cloud map based on the point cloud corresponding to the calibrated Tmid, and establish the mapping relationship between image pixels and three-dimensional point cloud by combining the pre-calibrated radar-camera extrinsic parameters. S3, the first laser point cloud and the second laser point cloud are superimposed and denoised to improve the point cloud density in the crack area; S4. The crack image is segmented using the trained crack segmentation model to obtain the crack mask. The crack mask region is then spatially searched and matched with the 3D point cloud to establish an accurate correspondence. S5 calculates the length, width, and depth parameters of the crack and generates a crack detection report. For scenarios where the sensor moving speed is less than or equal to a set threshold, the motion offset between T1 and T2 is compensated using the pre-integrated data from the inertial measurement unit before calculating Tmid.

2. The method for three-dimensional detection of building cracks based on image synchronization with dual point cloud as described in claim 1, characterized in that, In step S1, the specific process of establishing the time synchronization mechanism is as follows: The system controls the lidar to acquire the first frame of point cloud data and records T1 using a hardware timer; After completing the acquisition of the first frame of point cloud, the system keeps the sensor stationary and then sends a shooting trigger signal to the camera. After receiving the trigger signal, the camera completes image acquisition within the preset exposure delay time. After completing the crack image acquisition, the system controls the lidar again to acquire the second frame of point cloud data and records T2; The system calculates Tmid based on T1 and T2, and uses this Tmid to calibrate the crack image timestamp from the original Timg to Tmid in order to eliminate the drift error between the camera clock and the lidar clock.

3. The method for three-dimensional detection of building cracks based on image synchronization with dual point cloud as described in claim 2, characterized in that, In step S2, the construction process of the 3D point cloud map is as follows: The pose estimation is achieved by tightly coupling and fusing laser point cloud data with inertial measurement unit data using the lidar inertial odometry method. An incremental map management strategy is adopted to gradually add the scanned and registered point cloud data to the global map; The original point cloud is downsampled using a voxel grid filtering method to reduce the amount of data and computational complexity; An improved version of the iterative nearest point algorithm is used to match the point cloud of the current scan frame with the point cloud in the local map to estimate the pose transformation; The ray casting method is used to find the 3D points in the point cloud map that correspond to the image pixels.

4. The method for three-dimensional detection of building cracks based on image synchronization with dual point cloud as described in claim 3, characterized in that, The process of establishing the mapping relationship between image pixels and 3D point cloud is as follows: Normalized coordinates are obtained by projecting the pixel coordinates of the crack image onto the normalized plane using the camera intrinsic parameter matrix. The direction vector of the three-dimensional ray is obtained by transforming the normalized coordinates to the lidar coordinate system using the rotation matrix of the radar-camera extrinsic parameters; the radar-camera extrinsic parameters are pre-calibrated using a joint optimization method with a calibration board. Ray projection is performed on the point cloud map along the ray direction, and the three-dimensional point corresponding to the image pixel depth is quickly located by using a binary search method.

5. The method for three-dimensional detection of building cracks based on image synchronization with dual point cloud as described in claim 4, characterized in that, In step S3, the superposition denoising processing method is as follows: The point cloud registration algorithm is used to transform the second frame point cloud into the coordinate system of the first frame point cloud to achieve spatial alignment; Two frames of point clouds are merged into a higher-density composite point cloud using a point cloud fusion algorithm. Statistical filtering is used to identify and remove noise points. The Euclidean distance between each point and all points in its neighborhood is calculated. If the mean Euclidean distance between a point and its neighbors is greater than a set threshold, the point is identified as a noise point and removed. A pass-through filter is used to remove outliers in a certain direction from the point cloud; Conditional filtering is used to filter and remove points with excessively low reflection intensity or abnormal depth values ​​based on the attribute characteristics of the points.

6. The method for three-dimensional detection of building cracks based on image synchronization with dual point cloud as described in claim 5, characterized in that, In step S4, the crack segmentation model adopts a fully convolutional network architecture, which is an encoder-decoder structure. The encoder part uses a residual network as the backbone feature extraction network, which contains multiple residual stages. Each stage contains multiple residual units. The skip connection design of the residual units is used to alleviate the gradient vanishing problem of deep networks. The feature map resolution of each stage of the encoder decreases step by step, and the number of channels increases step by step. The decoder part gradually restores the spatial resolution of the feature map through transposed convolution operation, and fuses the feature maps of the corresponding layers of the encoder through skip connections. The decoder outputs a crack segmentation mask with the same resolution as the input image. The mask is a binary image, where a pixel value of one represents the crack region and a pixel value of zero represents the background region.

7. A method for three-dimensional detection of building cracks based on image synchronization using dual point cloud clips as described in claim 6, characterized in that, In step S5, the calculation method for the length, width, and depth parameters of the crack is as follows: Extract the coordinates of all three-dimensional points in the crack mask projection region to form a three-dimensional point set, calculate the Euclidean distance between each pair of points in the point set and take the maximum value as the crack length; Multiple sampling points are obtained by sampling the center line of the crack at equal intervals. The depth difference between the two edge points of the crack on both sides in the normal direction of each sampling point is extracted, and the mean of the depth difference of all sampling points is calculated as the average width of the crack. The crack surface points are fitted to a plane equation using the least squares method. The distances from all crack surface points to this plane are calculated, and the average value is taken as the crack depth.

8. A method for three-dimensional detection of building cracks based on image synchronization using dual point cloud clips as described in claim 7, characterized in that, In step S5, the specific method for motion offset compensation is as follows: The rotation increment and displacement increment are obtained by integrating the raw angular velocity and acceleration data of the inertial measurement unit during the time period from T1 to T2. Motion compensation is achieved by applying rotation and translation transformations to each 3D point in the second point cloud using rotation and displacement increments. After completing motion compensation, calculate the calibrated intermediate timestamp according to the correction formula; The pre-integration results of the inertial measurement unit are optimized and corrected online by combining visual odometry. A sliding window filter is used to fuse the pose estimation of visual odometry with the pre-integration results of the inertial measurement unit.

9. A method for three-dimensional detection of building cracks based on image synchronization using dual point cloud clips as described in claim 8, characterized in that, The online optimization and correction process is as follows: A continuous-time trajectory optimization method is used to globally adjust the historical poses and construct a pose graph optimization problem, where nodes are the sensor poses at each time step, and edges are the pre-integration constraints of the inertial measurement unit and the visual odometry constraints. High-precision sensor trajectories are obtained by solving the global optimal solution of the pose graph using a nonlinear optimization algorithm. Based on the optimized trajectory, distortion correction is performed on the point cloud of each frame to eliminate the point cloud deformation caused by motion. The distortion correction process uses a linear interpolation method to interpolate the pose according to the timestamp of each point and transform the point cloud coordinates to a unified reference time point.

Images