Crack detection and three-dimensional positioning method based on deep learning and SLAM technology

Abstract

The present invention discloses a crack detection and three-dimensional positioning method based on deep learning and SLAM technology, comprising the following steps: creating a data set, constructing a crack detection network model, and training and saving the optimal model; fusing data collected by an inertial measurement unit (IMU) and a camera KinectV2 using a tight coupling method with nonlinear optimization, and completing pose estimation in the tracking thread of a visual SLAM; judging each frame of real-time collected images in the tracking thread, screening out key frames, performing semantic segmentation on the key frames using a crack detection model, and extracting crack information; integrating the crack information and depth information collected by KinectV2 into the visual inertial SLAM framework, completing the construction of a dense point cloud map containing the crack information, and realizing three-dimensional positioning of the cracks. The method of the present invention realizes real-time detection of cracks and their positioning in a three-dimensional environment, while having high detection accuracy and good robustness.

Description

Technical Field

[0001] The present invention relates to a pavement crack detection and positioning method, and in particular to a crack detection and three-dimensional positioning method based on deep learning and SLAM technology. Background Art

[0002] Pavement cracks are the most common road hazard, making proper detection of pavement cracks crucial for road maintenance. Traditional pavement inspections require manual labor, which not only consumes significant time and effort but also suffers from low efficiency and safety. Therefore, research is needed to develop automated methods for detecting pavement cracks. Common automated detection algorithms are mostly based on deep learning networks. Deep learning-based crack detection algorithms will not only improve the efficiency of pavement maintenance, but also enhance the efficiency of related infrastructure maintenance, such as bridges, tunnels, and dams. This reduces inspection and maintenance costs and automates crack defect inspections, demonstrating significant application value and significance.

[0003] Invention patent 201911371906.3 discloses an image processing-based crack detection method and system. This method uses a pixel-based tracking algorithm to connect cracks and simultaneously analyzes the shape characteristics of the connected areas to filter out areas that do not meet the requirements. The model implemented by this method requires manual feature selection and is significantly affected by environmental factors, resulting in limited generalizability.

[0004] Invention patent 201911355501.0 discloses a crack detection network based on the Faster R-CNN network. This network can classify cracks from background and obtain the positioning results of the crack bounding box. However, the initial preprocessing of the image in this method ignores the interference of road noise and different light intensities, so the final crack detection accuracy is low, and the method will be greatly affected by interference.

[0005] Invention patent 202010247786.2 discloses a method for rapid identification of pavement cracks based on deep learning. The method uses a feature extraction network to map the road image to be detected to a coding space to obtain feature images of different sizes; for each size of the feature image, it is input into a crack detection network, and subjected to progressive feature fusion, preset convolution operation, attention enhancement, and subsequent preset convolution operation and upsampling to obtain a feature image with the same size as the road image and marked with crack areas, but no subsequent feature-based processing is performed on the cracks to quantify the cracks.

[0006] Invention patent 202010236401.2 discloses a crack detection model based on yolov5, which can output crack monitoring information, including the image prediction box, detection category and category confidence. At the same time, the loss function of the model is set as the weighted sum of the classification loss function, target loss function, regression loss function and cheap angle loss function, which improves the anti-interference ability and accuracy of crack detection. However, the network cannot obtain the specific location information of the crack.

[0007] In summary, crack detection based on deep learning networks can improve crack efficiency, but many current crack detection networks have high requirements for the number of training sets and image processing, and the network model is relatively complex. At the same time, they only obtain the position of the crack in the plane image, but do not obtain the three-dimensional position of the crack in the environment, which is not conducive to subsequent repair work. Summary of the Invention

[0008] The purpose of the present invention is to provide a crack detection and three-dimensional positioning method based on deep learning and SLAM technology, to solve the problem that many current crack detection networks have low model accuracy and efficiency, and cannot complete the three-dimensional positioning of cracks, and to complete the real-time pavement crack detection of mobile robots in the environment.

[0009] The technical solution for implementing the present invention is: a crack detection and three-dimensional positioning method based on deep learning and SLAM technology, comprising the following steps:

[0010] S1. Create a crack detection network dataset, build a crack detection network model, and train and save the optimal model;

[0011] S2, the data collected by the inertial measurement unit (IMU) and KinectV2 are fused using a tightly coupled method with nonlinear optimization, and pose estimation is completed in the tracking thread of the visual SLAM framework;

[0012] S3. In the tracking thread, each frame of the real-time collected image is judged, key frames are selected, and the key frames are semantically segmented using the crack detection model to extract crack information.

[0013] S4. Integrate the two-dimensional crack detection module obtained in step S3 and the depth information collected by KinectV2 into the visual inertial SLAM framework to complete the construction of a dense point cloud map containing crack information and realize the three-dimensional positioning of the crack.

[0014] Furthermore, the specific operations of step S1 are as follows:

[0015] (1) Collecting pavement crack images, preprocessing and labeling them;

[0016] (2) Based on the construction mode of the fully convolutional neural network, the encoding part consists of four groups of identical encoding blocks. Each encoding block consists of two 3×3 convolutional layers and one 1×1 convolutional layer. A residual module is added to each layer, and then enters a 2×2 maximum pooling layer. The decoding part of the network consists of four groups of identical decoding blocks. The feature image obtained after deconvolution is jump-connected with the feature image of the corresponding layer of the encoding part, and a dual-channel attention mechanism is added to the jump connection. A 1×1 convolutional layer is added to the last layer of the decoding part to achieve end-to-end crack segmentation and obtain a crack detection result image of the same size as the input image.

[0017] Furthermore, the specific operations of step S2 are as follows:

[0018] (1) Calibrate the KinectV2 camera and IMU separately to obtain their intrinsic parameters, and then jointly calibrate the two to obtain their transformation matrix and time difference;

[0019] (2) The IMU pre-integration model between consecutive frames is calculated through the IMU error model and motion model to solve the problem of misalignment between the camera and IMU acquisition information frequencies; and the gyroscope bias, gravity acceleration, and velocity are initialized to complete the visual-inertial joint initialization;

[0020] (3) The IMU pre-integration and visual information are tightly coupled for data fusion, and the pose estimation is achieved by a nonlinear optimization method based on sliding window marginalization for the visual reprojection error and IMU residual, and the optimized pose is obtained through the target optimization function.

[0021] Furthermore, the key frame screening rules in step S3 are as follows, and only one of them is required:

[0022] 1) At least 20 frames away from the last reference key frame;

[0023] 2) The current frame is at least 20 frames away from the last keyframe insertion or the local thread is idle;

[0024] 3) The current frame can track at least 50 feature points;

[0025] 4) The map points tracked in the current frame must be at least 80% more than those in the reference keyframe, ensuring low overlap;

[0026] Furthermore, the specific operations of step S4 are as follows:

[0027] (1) Through visual SLAM tracking, BA optimization and loop detection correction, key frames are selected. The color images and depth images corresponding to these key frames collected by KinectV2 are used to calculate the coordinates of each pixel in the camera coordinate system and the world coordinate system according to the camera imaging principle of KinectV2 to construct a dense point cloud map.

[0028] (2) The crack information obtained by semantic segmentation of all key frames in step 3 is mapped to the dense point cloud map through coordinate transformation. In the case of inconsistent semantic labels between consecutive key frames, Bayesian update is used. The specific method is as follows:

[0029]

[0030] Among them, the three-dimensional point is set to V d , Represents the current set of all key frames, l k Represents the category of a 3D voxel, Represents the independent distribution probability distribution of the 3D point on the semantic label set.

[0031] By updating the semantic information of the three-dimensional point cloud through the above formula, a globally consistent dense point cloud map with crack information is obtained, and the three-dimensional position information of the crack can be obtained through the point cloud.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] (1) The deep learning crack detection network constructed by the present invention simplifies the difficulty and complexity of crack detection and improves the accuracy and robustness of crack detection;

[0034] (2) The present invention fuses the measurement data of KinectV2 and IMU based on the sliding window tightly coupled back-end nonlinear optimization method, thereby completing the pose estimation, improving the positioning accuracy, and solving the problem that pure vision is prone to tracking loss and positioning failure in scenes with sparse feature points;

[0035] (3) The present invention combines visual-inertial SLAM technology with a two-dimensional crack detection network to construct a dense point cloud map containing crack information, solving the problem of lack of semantic information in the sparse point cloud map constructed in the original visual SLAM, and realizing three-dimensional positioning of cracks. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of crack detection and positioning of the present invention.

[0037] Figure 2 This is the deep learning network structure diagram of the present invention.

[0038] Figure 3This is a dual-channel attention gate diagram in the deep learning network of the present invention.

[0039] Figure 4 This is a flowchart for constructing a dense point cloud map based on visual SLAM in the present invention. DETAILED DESCRIPTION

[0040] The specific implementation of the present invention is described in detail below with reference to the accompanying drawings and implementation examples.

[0041] according to Figure 1 The present invention is a crack detection and three-dimensional positioning method based on deep learning and SLAM technology. The crack detection method comprises the following steps:

[0042] S1. Create an image training set, build a crack detection network model, and train and save the optimal model.

[0043] The specific operations of step S1 are as follows:

[0044] (1) Collect the original data and perform data expansion, and normalize the images, including cropping and grayscale conversion. To address the impact of uneven illumination intensity, histogram equalization and Gaussian bilateral filtering are used for image preprocessing;

[0045] (2) Annotate the preprocessed image, marking the cracked area in white and the non-cracked area in black, and obtain and save the mask image. The image dataset is divided into a training set and a test set with a ratio of 7:3. The training set contains an equal number of images with cracks and images without cracks.

[0046] (3) Build Figure 2 and Figure 3 The crack segmentation network shown in the figure includes four downsampling and four upsampling: each downsampling first uses 3×3 and 1×1 convolution kernels for convolution, then uses the ReLu activation function to activate the convolved image, and then cascades with the original input information. Downsampling is performed using a pooling of size 2×2, and the result is used as the input of the next layer of convolution. After this operation, the feature channel is expanded to twice that of the previous layer of the network; upsampling means that when the model enters the sixth layer, the output of the fifth layer is used as the gate signal, and the gate signal is first expanded to twice the original value through 1×1 deconvolution, and then input into the set AG module together with the output of the fourth layer of the model. After the data is processed by the AG module, the output signal is cascaded with the upsampled gate signal and output, entering the convolution layer with a convolution kernel size of 3×3. The output result is cascaded with the original input signal of the sixth layer as the input of the seventh layer, and so on. However, in the last layer of the network, a 1×1 convolution is used to map each 64-dimensional feature vector to the output layer.

[0047] The GDL loss function is selected as the loss function of the deep learning network. The conventional loss cross entropy function has the problem that it is good at segmenting image background pixels but is insensitive to crack pixels, which affects the accuracy of crack detection. The Adam optimization algorithm is used to update the model parameters.

[0048] Implement the aforementioned crack detection network in the PyTorch framework and train it using the selected training strategy and image training set. Use GPU mode for training. Most network hyperparameters, such as the learning rate and number of iterations, are set based on the specific training situation. Observe the loss and accuracy curves to obtain and save the optimal crack detection model. Verify the model's generalization and accuracy using the test set.

[0049] S2. The data collected by the inertial measurement unit (IMU) and KinectV2 are fused using a tight coupling method with nonlinear optimization, and pose estimation is completed in the tracking thread of visual SLAM.

[0050] The specific operations of step S2 are as follows:

[0051] (1) Use the calibration tools iai_kinect2 and kalibr_allan in ROS to calibrate KinectV2 and IMU respectively to obtain the intrinsic parameters; and perform a joint calibration of the two to obtain the conversion matrix and the time difference caused by the sampling frequency.

[0052] (2) The error model of IMU is as follows.

[0053]

[0054] in, Figure 2 shows the measurements of the accelerometer and gyroscope. a(t) and ω(t) represent their true values. b(t) refers to the slowly changing error generated by the sensor, known as random walk noise. η(t) represents rapidly changing white noise.

[0055] Substituting the IMU error model into the motion model, we get the following IMU complete motion model:

[0056]

[0057] Among them, R EB Represents the rotation matrix from the world coordinate system to the IMU coordinate system, a W 、v W 、p W Represent the acceleration, velocity, and translation in the world coordinate system respectively. i and j are two adjacent key frames, δφ ij ,δv ij ,δp ijRespectively represent the noise of IMU rotation, speed and position measurement, R i 、v i 、p i Represents the relative rotation matrix, velocity and translation of the i-th key frame respectively; R j 、v j 、p j Represents the relative rotation matrix, velocity and translation of the jth key frame respectively; Δt ij Represents the time difference between two moments i and j.

[0058] Perform pre-integration on adjacent key frames to obtain the pre-integration model as shown in formula (3).

[0059]

[0060] Among them, δφ i ,δv ij ,δp i They represent the noise of IMU's rotation, velocity, and position measurements respectively.

[0061] (3) Initialize the gyroscope bias, gravity acceleration, and velocity to reduce the cumulative error of the IMU.

[0062] (4) The residual model of the IMU and the reprojection error model of the camera are shown in Equations (4) and (5), and a nonlinear optimization model is established based on this, as shown in Equation (6).

[0063]

[0064] in, is the residual model of the inertial measurement unit, where X is the variable to be optimized, is the IMU pre-integrated value, e p 、e q 、e v 、e ba 、e bg are the position, velocity, attitude, gyroscope bias, and accelerometer bias in the pre-integration of the inertial measurement unit, b is the random walk noise, and β is the white noise; is the visual reprojection error of the entire system, ξ i is the Lie algebra corresponding to the camera pose, m j is a 3D map point, p ij is the pixel point corresponding to the 3D map point in the image.

[0065] To address the problem that the computational complexity of the optimization model increases dramatically as the system runs, a sliding window approach is used for optimization. That is, only the constraint relationship between the removed keyframes and the remaining frames is retained in the sliding window, rather than directly removing the old keyframes. The retained constraint relationship is not used for optimization, thus reducing the computational complexity. Therefore, the present invention establishes an optimization objective function as shown in formula (7) to perform pose optimization, where the three terms represent the prior information, the visual reprojection error, and the IMU residual, respectively.

[0066]

[0067] Where B represents the set of all IMU measurements, k represents the kth image, and H p The Hessian matrix representing the prior information, γ represents the Jacobian matrix of each optimization variable

[0068] S3. In the tracking thread, each frame of the real-time image is judged, key frames are selected, and the key frames are semantically segmented using the crack detection model to extract crack information.

[0069] The keyframe filtering rules are as follows, and only one of them is required:

[0070] 1) At least 20 frames away from the last reference key frame;

[0071] 2) The current frame is at least 20 frames away from the last keyframe insertion or the local thread is idle;

[0072] 3) The current frame can track at least 50 feature points;

[0073] 4) The map points tracked in the current frame must be at least 80% more than those in the reference keyframe, ensuring low overlap;

[0074] S4. Integrate the two-dimensional crack detection module obtained in step 4 and the depth information collected by KinectV2 into the visual SLAM framework to complete the construction of a dense point cloud map containing crack information and realize the three-dimensional positioning of the crack.

[0075] The specific operations of step S4 are as follows:

[0076] (1) According to Figure 4 The color image and depth image collected by KinectV2 are used as the sensor input of visual SLAM, and then the color image and depth image corresponding to the key frames that meet the rules are screened out through tracking, BA optimization and loop detection.

[0077] (2) Each pixel in the depth map contains its two-dimensional pixel coordinates (u, v) on the color map and its depth value d. Based on the camera's imaging principle and the parameters obtained from the Kinectv2 camera calibration, the coordinates of the pixel are converted to the camera coordinate system using the formula, and then converted to the world coordinate system using the formula (8). The dense point cloud map in the PCD format is obtained by processing using the functions in the PCL library.

[0078]

[0079] Among them, C is the camera intrinsic parameter matrix, R and t are the camera's rotation matrix and translation vector, [x, y, z] T Represents the point cloud coordinates in the world coordinate system.

[0080] (3) In step 3, crack detection is performed on the keyframes. Then, the coordinates of the map points corresponding to the feature points of the segmented cracks are converted to the world coordinate system through the coordinate transformation relationship. Due to the instability of sensor data acquisition, the semantic labels of cracks between consecutive keyframes may be inconsistent, resulting in conflicts when semantic labels are fused. A progressive semantic label association method based on Bayesian estimation is used.

[0081] Assume that the key frame K is currently being processed t , the three-dimensional point is set to V d , then the set of all key frames is Then, through Bayesian updating, we can know the distribution of the semantic label of this point l k As shown below:

[0082]

[0083] Then, by using the Markov assumption, we can get formula (10):

[0084]

[0085] in, Since P(l k ) is quantitative and does not change with time, so the regularization factor can be ignored The semantic information of the 3D point cloud is updated when a new key frame arrives through formula (11).

[0086]

[0087] The two-dimensional semantic labels of multiple key frames are associated through Bayesian updating, and then transferred to a three-dimensional point cloud through the coordinate transformation relationship in the dense reconstruction thread, so as to obtain a three-dimensional crack semantic map with consistent global semantic labels.

[0088] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A crack detection and three-dimensional positioning method based on deep learning and SLAM technology, characterized in that: Specifically include the following steps: S1. Build a crack detection network model and train it using the data set to obtain the optimal model. S2. The data collected by the inertial measurement unit and the camera are fused using a tight coupling method with nonlinear optimization, and pose estimation is completed in the tracking thread of the visual SLAM. S3. In the tracking thread, each frame of the real-time image is judged, key frames are selected, and the key frames are semantically segmented using the crack detection network model to extract crack information. S4, integrating the crack information obtained in step S3 and the data collected by the camera into the visual SLAM framework, constructing a dense point cloud map containing the crack information, and completing the three-dimensional positioning of the crack; The step S1 specifically includes: Step S1-1, collecting raw data and performing data expansion to construct a data set, normalizing the data set, and performing image preprocessing using histogram equalization and Gaussian bilateral filtering; Step S1-2: annotate the preprocessed image, mark the crack area in white and the non-crack area in black, obtain the mask image and save it, and divide the data set into a training set and a test set; Step S1-3, building a crack detection network model, including four downsampling and four upsampling: each downsampling first uses 3×3 and 1×1 convolution kernels for convolution, then uses the ReLu activation function to activate the convolved image, and then cascades with the original input information. Downsampling is performed using a pooling of 2×2, and the result is used as the input of the next layer of convolution; upsampling means that when the model enters the sixth layer, the output of the fifth layer is used as the gate signal, and the gate signal is first expanded to twice the original value through 1×1 deconvolution, and then input into the set AG module together with the output of the fourth layer of the model. After the data is processed by the AG module, the output signal is cascaded with the upsampled gate signal and output, entering the convolution layer with a convolution kernel size of 3×3. The output result is cascaded with the original input signal of the sixth layer as the input of the seventh layer, and so on. In the last layer of the network, a 1×1 convolution is used to map each 64-dimensional feature vector to the output layer; Step S1-4, iteratively training the crack detection network model based on the training set and verifying it through the test set until an optimal model that meets the set values ​​is obtained; The ratio of the training set to the test set is 7:3, and the number of images with cracks and images without cracks contained in the training set is equal; The step S2 specifically includes: S2-1, calibrate the camera and inertial measurement unit separately to obtain their internal parameters, and then jointly calibrate the two to obtain the transformation matrix and time difference; S2-2, calculate the pre-integration model between consecutive frames through the error model and motion model of the inertial measurement unit, and initialize the gyroscope bias, gravity acceleration, and velocity to complete the visual-inertial joint initialization; S2-3, the pre-integration model is pre-integrated and then fused with the visual information in a tightly coupled manner. The visual reprojection error and the inertial measurement unit residual are subjected to pose estimation using a nonlinear optimization model based on sliding window marginalization, and the optimized pose is determined by the target optimization function. The pre-integration model is: Among them, i and j are two adjacent key frames, δφ ij ,δv ij ,δp ij They represent the noise of the rotation, speed and position measurement of the inertial measurement unit, R i 、v i 、p i Represents the relative rotation matrix, velocity and translation of the i-th key frame respectively; R j 、v j 、p j Represents the relative rotation matrix, velocity and translation of the jth key frame respectively; Δt ij Represents the time difference between two moments i and j; The nonlinear optimization model is: in, is the residual model of the inertial measurement unit, where X is the variable to be optimized, is the pre-integrated value of the inertial measurement unit, e p 、e q 、e v 、e ba 、e bg are the position, velocity, attitude, gyroscope bias, and accelerometer bias in the pre-integration of the inertial measurement unit, b is the random walk noise, and β is the white noise; is the visual reprojection error of the inertial measurement unit, ξ i is the Lie algebra corresponding to the camera pose, m j is a 3D map point, p ij is the pixel point corresponding to the 3D map point in the image; The objective optimization function is: Where B represents the set of all inertial measurement unit measurements, k represents the kth image, and H p The Hessian matrix representing the prior information, γ p , γ B Represents the Jacobian matrix of each optimization variable; The screening rule for filtering out key frames in step S3 is to satisfy one of the following rules: 1) At least 20 frames away from the last reference key frame; 2) The current frame is at least 20 frames away from the last keyframe insertion or the local thread is idle; 3) The current frame can track at least 50 feature points; 4) The 3D map points tracked in the current frame must be at least 80% more than those in the reference keyframe, ensuring low overlap; The step S4 specifically includes: S4-1, perform dense point cloud mapping on the color map and depth map corresponding to the key frames selected by visual SLAM tracking, local BA optimization and loop correction to obtain a dense point cloud map; S4-2, the crack information obtained by semantic segmentation of all key frames in step 3 is mapped to the dense point cloud map through coordinate transformation relationship, and Bayesian update is used to address the inconsistency of semantic labels between consecutive key frames, namely: Among them, the three-dimensional map point is set to V d , Represents the current set of all key frames, l k Represents the category of a 3D voxel, Represents the independent distribution probability distribution of the 3D map point on the semantic label set; When a new keyframe arrives, The semantic information of the 3D map point cloud is updated to obtain a globally consistent dense point cloud map with crack information, and the 3D position information of the crack is obtained through the dense point cloud map.

2. The crack detection and three-dimensional positioning method based on deep learning and SLAM technology according to claim 1 is characterized in that: The crack detection network model is built based on a fully convolutional neural network, a residual module, and an attention mechanism. A 1×1 convolution layer is added to the convolution group of each layer, and a residual module is added at the same time. A dual-channel attention mechanism is added to the jump connection part of encoding and decoding.

Images