Crack detection and three-dimensional positioning method based on SLAM and image processing

Abstract

This invention discloses a crack detection and three-dimensional positioning method based on SLAM and image processing. The method includes: using a mobile robot to collect road surface images in real time, transmitting them to an image processing system on a host computer to detect road cracks and quantify their lengths; improving the visual feature extraction method of the SLAM framework, fusing data from a Kinect V2 depth camera and an inertial measurement unit, and integrating this fusion into the visual SLAM framework through tight coupling to estimate the position and posture of the mobile robot; extracting key frames from the continuous frame images collected during the robot's motion according to rules, combining the images from the color camera and depth camera with the SLAM backend's pose estimation to obtain the position information of the crack's surrounding environment; and combining a point cloud library with the visual pose information to generate a dense point cloud map of the crack's surrounding environment. This method can monitor crack information and locate it in real time with high accuracy and strong robustness.

Description

Technical Field

[0001] The present invention relates to a crack detection and three-dimensional positioning method for a road, and in particular to a crack detection and three-dimensional positioning method based on SLAM and image processing. Background Art

[0002] Road cracks are one of the most common pavement defects. Once they appear, they pose a serious safety hazard, necessitating prompt road maintenance. Currently, pavement inspection is primarily performed manually, requiring knowledgeable personnel, is time-consuming and labor-intensive, and carries certain safety risks. Therefore, automated methods for detecting road cracks are needed. Common automated crack detection methods combine image processing and deep learning. Deep learning-based crack detection algorithms are highly efficient and fast. Computer vision technologies are used in road inspection, requiring visual sensors to capture road images. However, drones often capture large images, making them unsuitable for crack location research. Therefore, robots equipped with cameras are used to capture crack images.

[0003] Invention Patent 201710434581.3 discloses a pavement crack detection method based on image processing. This method uses image processing and image segmentation to extract image ossification and calculates crack information parameters based on pixel distance. This method requires manual extraction of crack features, has low generalization performance, and fails to achieve three-dimensional crack localization.

[0004] Invention patent 201911371906.3 discloses an image processing-based crack detection method and system. This method uses a pixel-based tracking algorithm to connect preprocessed crack images and analyze the connected image features. This method still requires manual feature design, is easily affected by surrounding environmental information, and has low generalization performance.

[0005] Invention patent 201911355501.0 discloses a crack detection network based on the Faster R-CNN network. This method uses the Faster R-CNN network to detect cracks and locate bounding boxes, but this method does not effectively remove road noise and light intensity interference, and the crack detection accuracy is not high.

[0006] Invention patent 202010236401.2 discloses a crack detection model based on yolov5, which can obtain the detection frame, detection category and category confidence of the crack. This method improves the accuracy of crack detection, but the real-time performance on mobile devices needs to be further improved, and this method does not obtain the three-dimensional position information of the crack.

[0007] In summary, current crack detection methods mostly use image processing and deep learning methods. The network model is relatively complex, and the crack location obtained is limited to the two-dimensional plane. The three-dimensional position information is not obtained, which is not conducive to the subsequent search and repair of the crack location. Summary of the Invention

[0008] The purpose of the present invention is to provide a crack detection and three-dimensional positioning method based on SLAM and image processing, which solves the problems of low generalization and low detection accuracy of crack detection, and at the same time adds a three-dimensional positioning function of the crack, which can realize real-time crack detection during robot movement.

[0009] The technical solution for implementing the present invention is as follows: In a first aspect, the present invention provides a crack detection and three-dimensional positioning method based on SLAM and image processing, comprising the following steps:

[0010] S1: The mobile robot is equipped with a depth camera to collect road images in real time, and the image processing system of the host computer detects crack information in the color image;

[0011] S2, based on the visual sensor, integrates the inertial measurement unit, fuses the data of the two through edge-based tight coupling, inputs the SLAM algorithm framework and completes the robot's pose estimation;

[0012] S3, improves the feature extraction method in visual SLAM, judges each frame of real-time collected images and filters out key frames, and inputs them into the image processing system to extract crack information;

[0013] S4, fusing the crack information extracted in step S3 with the depth camera data, and constructing a dense point cloud map containing the crack information through spatial matrix conversion to achieve three-dimensional positioning of the crack.

[0014] Furthermore, the image processing system of step S1 specifically includes:

[0015] Step S1-1: grayscale the collected road surface image and define a sliding window. The window slides from left to right and from top to bottom on the image. The window size is adaptively adjusted based on experimental results.

[0016] Step S1-2: Count the total grayscale values ​​of the image in each window to obtain the grayscale distribution map of all windows. By setting an adaptive threshold, the pixel values ​​of windows above the threshold are set to 0, and all windows below the threshold are defined as crack candidate areas.

[0017] Step S1-3, performing improved median filtering and image enhancement on the area to be detected, and then performing an adaptive thresholding operation to perform binary segmentation to obtain a binary image of the area to be detected;

[0018] Step S1-4: Performing a morphological dilation operation on the binary image to extract the skeleton of the crack binary image and quantifying the crack length.

[0019] Furthermore, the steps of the crack median filtering method for the area to be detected are as follows:

[0020] (1) Traverse the pixel points in the area to be detected and find the target pixel point I ij Difference between the grayscale value of each pixel in the eight neighborhoods and the target pixel;

[0021] (2) Take the three neighboring pixels with the smallest absolute value, and use I ij The direction of the neighboring pixel point is the axis extending outward, and three pixel points I are obtained. p ,I q ,I h ;

[0022] (3) Take the pixels and I in the eight neighborhoods p ,I q ,I h For three pixels, take the median value from small to large according to the grayscale value and replace it with the value of the target pixel.

[0023] Furthermore, in the binary segmentation using adaptive thresholding, the threshold setting method is: statistically analyzing the grayscale value distribution of the area to be detected, and adaptively setting the segmentation threshold according to the grayscale distribution histogram.

[0024] Furthermore, the quantitative processing method of crack length is:

[0025] Use Zhang's parallel thinning algorithm to obtain the crack skeleton, obtain the skeleton image at the pixel level, and obtain the pixel grayscale value on the skeleton line:

[0026]

[0027] Where (x, y) represents the pixel coordinates on the skeleton line, I(x, y) represents the pixel value at (x, y), and the number of pixels with a value of 1 on the skeleton line, n, is counted to calculate the crack length L:

[0028]

[0029] in, Represents the distance between adjacent pixels; (x i ,y i ) represents the coordinates of the i-th point on the skeleton line, (x i+1 ,y i+1 ) represents the coordinates of the i+1th point on the skeleton line.

[0030] Furthermore, the step S2 specifically includes:

[0031] S2-1, calibrate the camera and inertial measurement unit separately and calibrate them jointly to obtain the transformation matrix between them;

[0032] S2-2, build a model for the inertial measurement unit and perform pre-integration processing on the inertial measurement unit based on the modeling results; then initialize the visual camera and the inertial measurement unit together, mainly initializing the gyroscope bias and gravity acceleration;

[0033] S2-3, through the edge optimization method based on sliding window, the inertial sensor information and visual information are tightly coupled, the visual reprojection error and the inertial measurement unit observation residual are calculated, and they are merged into the sliding window. A nonlinear optimization model is established according to the optimization vector and the pose is estimated.

[0034] Furthermore, the SLAM feature extraction method in step S3 extracts ORB feature corner points on the image and uses the Beblid descriptor to match the feature points; when performing key frame screening, one of the following conditions must be met: (1) the interval between the last global relocalization is at least 20 frames; (2) the current key frame must track at least 50 map points; (3) the feature points tracked by the current key frame must contain less than 80% of the feature points of the previous key frame.

[0035] Furthermore, step S4 specifically includes: obtaining the three-dimensional position information of the crack in the dense map through spatial coordinate transformation of the crack key frame obtained through crack detection; after the key frame is integrated with SLAM, back-end optimization and loop correction, its color information and depth information are integrated to obtain a dense point cloud map with crack information.

[0036] In a second aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the program.

[0037] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] (1) The crack image processing system constructed by the present invention improves the accuracy of crack detection while ensuring the real-time performance of the mobile terminal, and has high robustness and generalization ability.

[0040] (2) Based on the visual sensor, the present invention integrates the data of the inertial measurement unit and tightly couples the two data through sliding window marginalization, thereby optimizing the pose estimation and improving the positioning accuracy, which to a certain extent makes up for the problem that visual features are prone to positioning failure.

[0041] (3) The present invention combines crack detection and positioning, obtains a dense point cloud map containing crack information through SLAM technology, and realizes three-dimensional reconstruction of cracks containing semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of crack detection and three-dimensional positioning of the present invention.

[0043] Figure 2 Schematic diagram of the image crack detection system of the present invention.

[0044] Figure 3 This is a schematic diagram of the improved median filter of the present invention.

[0045] Figure 4 Flowchart for crack location structure. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present invention clearer, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings and implementation examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0047] according to Figure 1 The present invention is a crack detection and three-dimensional positioning method based on SLAM and image processing. The crack detection method comprises the following steps:

[0048] S1, use the depth camera to collect the road crack image, input it into the image processing system of the host computer for crack detection, the crack detection process is as follows Figure 2 .

[0049] (1) A mobile robot equipped with a depth camera is used to acquire images. The images acquired by the camera include color information and depth information. The color image is grayscaled to obtain the grayscale image to be detected.

[0050] (2) Define a sliding window, start sliding from the upper left corner of the grayscale image, set the grayscale value sum of the entire image as the threshold, and count the total grayscale value of the grayscale image in the current window. When the statistical value is higher than the threshold, set all pixel values ​​in the window to 0. When it is lower than the threshold, set the window as a crack candidate area.

[0051] (3) Perform median filtering on all crack candidate areas. The specific implementation method is as follows: Figure 3 , traverse the candidate window, take each pixel as the center, make the difference between the pixel values ​​in its eight neighborhoods and the center value, and extend the three neighborhoods with the smallest absolute value outward to obtain the three extended pixel points I p ,I q ,Ih ,After arranging the eight pixels in the eight neighborhoods and the three extended pixels according to their grayscale values, ,the median value is taken to replace the value of the central pixel. ,Then image enhancement is used to process the crack candidate area so that the contrast between the crack and the background is deepened.

[0052] (4) Count the grayscale values ​​of the crack candidate area and generate a grayscale histogram. According to the distribution of different grayscale values ​​in the histogram, the threshold of binary segmentation is adaptively set to convert the candidate area into a binary image, where the part below the threshold is set to 1 and the part above the threshold is set to 0.

[0053] (5) Perform morphological operations on the binary image in the window to remove residual noise points, obtain the extracted crack image, and quantify the crack length. The processing method is:

[0054] Use Zhang's parallel thinning algorithm to obtain the crack skeleton, obtain the skeleton image at the pixel level, and obtain the pixel grayscale value on the skeleton line:

[0055]

[0056] Where (x, y) represents the pixel coordinates on the skeleton line, I(x, y) represents the pixel value at (x, y), and the number of pixels with a value of 1 on the skeleton line, n, is counted to calculate the crack length L:

[0057]

[0058] in, Represents the distance between adjacent pixels; (x i ,y i ) represents the coordinates of the i-th point on the skeleton line, (x i+1 ,y i+1 ) represents the coordinates of the i+1th point on the skeleton line.

[0059] S2, according to Figure 4 The data collected by the inertial measurement unit and the visual camera are tightly coupled and fused, and the position and pose estimation of the sensor data information is completed using visual SLAM.

[0060] The specific operation of S2 is as follows:

[0061] (1) Use the calibration tool provided by the robot operating system to calibrate KinectV2 and IMU separately, then fix the two sensors together and perform joint calibration to obtain the conversion matrix between the two.

[0062] (2) First, the IMU is modeled. The error model of the IMU is as shown in Equations (3) and (4):

[0063] w m =w+bg +η g (3)

[0064] a m =a+b a +η a (4)

[0065] Among them, w m 、a m represents the measurement data of the gyroscope and accelerometer, w and a represent the real data of the two, b represents the slowly changing random walk error generated by the sensor, and η represents the rapidly changing Gaussian white noise error.

[0066] The IMU kinematic model is as shown in equations (5)-(7):

[0067]

[0068]

[0069]

[0070] Where p, v, and q represent the system position, velocity, and rotation at the corresponding moment. a, v, and g represent the acceleration, angular velocity, and gravitational acceleration. b represents the offset, and Δt represents the time interval.

[0071] Perform pre-integration operation between adjacent IMU measurement frames to obtain the pre-integration model as shown in equations (8)-(10):

[0072]

[0073]

[0074]

[0075] Among them, i,i+1 represent two adjacent key frames, R i ,v i ,p i Represents the relative rotation matrix, velocity and translation of the i-th key frame; R i+1 ,v i+1 ,p i+1 represents the relative rotation matrix, velocity and translation of the i+1th key frame; Δt represents the time difference between two key frames. g , b a is zero bias.

[0076] (3) The cumulative error of inertial measurement is reduced by initializing the gyroscope bias and gravity acceleration.

[0077] (4) Establish the IMU observation residual model and the camera reprojection error model as shown in equations (11)-(12):

[0078]

[0079]

[0080] in, represents the observation residual of the inertial measurement unit, χ is the variable to be optimized, Represents the IMU pre-integrated values ​​of the gyroscope and accelerometer bias within two keyframes. Represents the displacement variation error, Indicates the speed change error, Indicates the angle change error; represents the visual reprojection error of the system, represents the lth feature point observed, Represents the back projection from the pixel position to the unit vector through the camera intrinsic parameters;

[0081] As time goes by, the complexity of the state variables at the previous moment will also increase, causing the subsequent computational complexity to increase. Therefore, a marginalization approach is adopted for processing. The specific implementation method is based on a sliding window. When removing the pose, the constraint relationship between the previous and next image frames is added as a priori conditions to the optimization problem, as shown in Equation (13). This method can avoid losing important information. If the penultimate frame is a key frame, the earliest position information is moved out of the window, and the sensor data information associated with the earliest frame is marginalized. The first old key frame and measurement information are also marginalized. Conversely, only the visual information of the penultimate frame is deleted to ensure the consistency of the IMU pre-integration and the sparsity of the system.

[0082]

[0083] Among them, ||r p -H p χ|| 2 represents the marginalized residual, represents the visual reprojection error, represents the IMU residual, ρ is the robust kernel function, χ represents the optimized state vector in the sliding window, B represents the inertial measurement set, k represents the kth image, H p represents the Hessian matrix containing prior information, and γ represents the Jacobian matrix to be optimized.

[0084] S3 improves the feature extraction method in visual SLAM, replacing the brief descriptor with the Beblid descriptor and using a combination of ORB and Beblid to extract image feature points and perform matching. Each frame of real-time image acquisition is judged and key frames are selected, which are input into the image processing system to extract crack information. The rules for key frame selection must meet the following requirements: (1) the interval between the last global relocalization and the last key frame must be at least 20 frames; (2) the current key frame must track at least 50 map points; and (3) the feature points tracked by the current key frame must contain less than 80% of the feature points of the previous key frame.

[0085] S4, the crack detection thread obtained in the above steps is integrated with the image depth information collected by the visual camera, and a dense point cloud map containing crack information is constructed through the PCL point cloud library, thereby realizing the crack positioning function.

[0086] The specific operations of step S4 are:

[0087] (1) According to Figure 3 The input of visual SLAM is the color and depth information collected by KinectV2 and the data information collected by IMU. Through the SLAM tracking thread, backend optimization thread and loop correction thread, the key frame image that meets the requirements is obtained.

[0088] (2) The key frame depth image contains the depth value of each pixel. The position of the key frame in the world coordinate system is obtained through the camera parameters obtained previously and the conversion from the camera coordinate system to the world coordinate system. The dense point cloud is drawn through the function in the PCL point cloud library to obtain a dense point cloud map with crack information.

[0089] Regarding the technical implementation in this field, the details of the above exemplary embodiments merely represent the technical solutions of the present invention and are not intended to be limiting. For the technical solutions of the above examples, those skilled in the art may make equivalent substitutions for some of the technical features, provided that such substitutions do not depart from the spirit and scope of the technical solutions of the present invention. Any reference numerals in the claims should not be construed as limiting the claims.

Claims

1. A crack detection and three-dimensional positioning method based on SLAM and image processing, characterized in that: The following steps are involved: S1: The mobile robot is equipped with a depth camera to collect road images in real time, and the image processing system of the host computer detects crack information in the color image; S2 integrates the inertial measurement unit with the visual sensor, fuses the data of the two through edge-based tight coupling, inputs the SLAM algorithm framework and completes the robot's pose estimation; specifically, it includes: S2-1, calibrate the camera and inertial measurement unit separately and calibrate them jointly to obtain the transformation matrix between them; S2-2, modeling the inertial measurement unit and performing pre-integration processing on the inertial measurement unit based on the modeling results; then jointly initializing the visual camera and the inertial measurement unit, that is, initializing the gyroscope bias and gravity acceleration; S2-3, tightly couples inertial sensor information and visual information through edge optimization based on sliding windows, calculates visual reprojection error and inertial measurement unit observation residual, merges them into the sliding window, establishes a nonlinear optimization model based on the optimization vector, and estimates the pose; The kinematic model of the inertial measurement unit is as follows: Where p, v, q represent the system position, velocity, and rotation information at the corresponding moment; a, g represent acceleration and gravity acceleration; b represents the offset, and Δt represents the time interval; The pre-integration process is as shown in equations (8)-(10): Among them, i,i+1 represent two adjacent key frames, R i ,v i ,p i Represents the relative rotation matrix, velocity and translation of the i-th key frame; The observation residual and the reprojection error are as shown in equations (11)-(12): in, represents the observation residual of the inertial measurement unit, χ is the variable to be optimized, Represents the IMU pre-integrated value of the gyroscope and accelerometer zero bias within two key frames; Represents the displacement variation error, Indicates the speed change error, Indicates the angle change error; represents the visual reprojection error of the system, represents the lth feature point observed, Represents the back projection from the pixel position to the unit vector through the camera intrinsic parameters; The nonlinear optimization model is: Among them, ||r p -H p χ|| 2 represents the marginalized residual, represents the visual reprojection error, represents the IMU residual, ρ is the robust kernel function, χ represents the optimized state vector in the sliding window, B represents the inertial measurement set, k represents the kth image, H p represents the Hessian matrix containing prior information, and γ represents the Jacobian matrix to be optimized; S3, improves the feature extraction method in visual SLAM, judges each frame of real-time collected images and filters out key frames, and inputs them into the image processing system to extract crack information; The feature extraction method of SLAM is to extract ORB feature corner points on the image and match the feature points using Beblid descriptors; when performing key frame screening, one of the following conditions must be met: (1) the interval between the last global relocalization is at least 20 frames; (2) the current key frame must track at least 50 map points; (3) the feature points tracked by the current key frame must contain less than 80% of the feature points of the previous key frame; S4, fusing the crack information extracted in step S3 with the depth camera data, and constructing a dense point cloud map containing the crack information through spatial matrix conversion to achieve three-dimensional positioning of the crack.

2. The crack detection and three-dimensional positioning method based on SLAM and image processing according to claim 1 is characterized in that: The image processing system of step S1 specifically includes: Step S1-1: grayscale the collected road surface image and define a sliding window. The window slides from left to right and from top to bottom on the image. The window size is adaptively adjusted based on experimental results. Step S1-2: Count the total grayscale values ​​of the image in each window to obtain the grayscale distribution map of all windows. By setting an adaptive threshold, the pixel values ​​of windows above the threshold are set to 0, and all windows below the threshold are defined as crack candidate areas. Step S1-3, performing improved median filtering and image enhancement on the area to be detected, and then performing an adaptive thresholding operation to perform binary segmentation to obtain a binary image of the area to be detected; Step S1-4: Performing a morphological dilation operation on the binary image to extract the skeleton of the crack binary image and quantifying the crack length.

3. The crack detection and three-dimensional positioning method based on SLAM and image processing according to claim 2 is characterized in that: The steps of the median filtering method for cracks in the area to be detected are as follows: (1) Traverse the pixel points in the area to be detected and find the target pixel point I ij Difference between the grayscale value of each pixel in the eight neighborhoods and the target pixel; (2) Take the three neighboring pixels with the smallest absolute value, and use I ij The direction of the neighboring pixel point is the axis extending outward, and three pixel points I are obtained. p ,I q ,I h ; (3) Take the pixels and I in the eight neighborhoods p ,I q ,I h For three pixels, take the median value from small to large according to the grayscale value and replace it with the value of the target pixel.

4. The crack detection and three-dimensional positioning method based on SLAM and image processing according to claim 2 is characterized in that: In the binary segmentation using adaptive thresholding, the threshold setting method is: statistically analyzing the grayscale value distribution of the area to be detected, and adaptively setting the segmentation threshold according to the grayscale distribution histogram.

5. The crack detection and three-dimensional positioning method based on SLAM and image processing according to claim 2 is characterized in that: The quantitative processing method for crack length is: Use Zhang's parallel thinning algorithm to obtain the crack skeleton, obtain the skeleton image at the pixel level, and obtain the pixel grayscale value on the skeleton line: Where (x, y) represents the pixel coordinates on the skeleton line, I(x, y) represents the pixel value at (x, y), and the number of pixels with a value of 1 on the skeleton line, n, is counted to calculate the crack length L: in, Represents the distance between adjacent pixels; (x i ,y i ) represents the coordinates of the i-th point on the skeleton line, (x i+1 ,y i+1 ) represents the coordinates of the i+1th point on the skeleton line.

6. The crack detection and three-dimensional positioning method based on SLAM and image processing according to claim 1 is characterized in that: The step S4 specifically includes: obtaining the crack keyframes obtained through crack detection, obtaining the three-dimensional position information of the cracks in the dense map through spatial coordinate transformation; fusing the keyframes with their color information and depth information after SLAM fusion, back-end optimization and loop correction, and obtaining a dense point cloud map with crack information.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images