Road crack recognition method based on multi-stage registration mapping of point cloud-RGB heterogeneous images

Abstract

The application relates to a pavement crack identification method based on point cloud-RGB heterogenous image multi-stage registration mapping. The method comprises the following steps: collecting pavement point cloud data and pavement image data for time and space synchronization, creating a projection image pair for feature extraction, obtaining a local feature descriptor, matching feature points in the local feature descriptor, obtaining an actual matching point pair, solving the parameters of a direct linear transformation equation according to the actual matching point pair, obtaining the mapping relationship between the RGB image pixel coordinates and the three-dimensional point cloud coordinates, traversing the coordinates of each point in the pavement image data, assigning the mapping relationship depth information to the pavement image data, obtaining a point cloud projection image, generating a depth image from the point cloud projection image, labeling cracks and backgrounds, constructing a data set, training a crack identification model using the data set, obtaining a trained crack identification model, and identifying pavement cracks, thereby improving the efficiency and accuracy of automatic crack identification.

Description

Technical Field

[0001] This application relates to the field of road engineering inspection and maintenance technology based on image recognition, and in particular to a method for identifying pavement cracks based on multi-level registration mapping of point cloud-RGB heterogeneous images. Background Technology

[0002] With the continuous increase in highway mileage and the mounting pressure of maintenance demands, the requirements for road inspection are also becoming increasingly stringent. Automation, intelligence, and precision have become the main trends in the development of pavement inspection technology. Cracks, as the most common type of pavement distress, require timely and accurate detection and repair, which significantly improves maintenance efficiency and maintains pavement performance. Driven by this demand, automatic crack recognition technology based on pavement images has made great strides.

[0003] However, single visible light images or three-dimensional depth images are affected by shadows, uneven lighting, or false depth, and cannot fully reveal information such as the spatial texture or depth of road surface cracks. False results are common, which seriously affects the efficiency and accuracy of road surface crack detection.

[0004] In recent years, with the continuous improvement of lidar equipment, the fusion technology of 3D point clouds and visible light images has become an increasingly popular research topic. 3D point clouds possess rich spatial and reflectivity information and are not easily affected by external lighting, but they have low resolution and lack realistic texture and color information. Conversely, optical images can obtain realistic texture and color information and have high resolution, but they are greatly affected by lighting during acquisition and cannot acquire spatial distance information.

[0005] Therefore, the efficiency and accuracy of current automatic crack identification based on road surface images are relatively low. Summary of the Invention

[0006] Therefore, it is necessary to provide a road surface crack identification method based on multi-level registration mapping of point cloud-RGB heterogeneous images that can improve the efficiency and accuracy of automatic crack identification, in order to address the above-mentioned technical problems.

[0007] A method for identifying road surface cracks based on multi-level registration mapping of point cloud-RGB heterogeneous images, the method comprising:

[0008] The road surface point cloud data and road surface image data are collected using a lidar and camera mounted on a drone. The road surface point cloud data includes point coordinate information and time information, and the road surface image data includes time information and size information.

[0009] The road surface point cloud data and the road surface image data are time-synchronized using a frame-synchronized acquisition method, and the road surface point cloud data and the road surface image data are spatially synchronized using a spatial transformation matrix to create mutually matching projection image pairs.

[0010] Based on the significant nonlinear differences in grayscale distribution between two-dimensional and three-dimensional images, while maintaining structural consistency, feature extraction is performed on the projected image pairs to obtain local feature descriptors for the projected image pairs based on enhanced structural consistency.

[0011] Feature points are extracted from the local feature descriptor. Nearest neighbor distance ratio detection is used as the matching strategy. The FAST feature point detector is used to match the feature points to obtain actual matching point pairs.

[0012] The mapping relationship between RGB image pixel coordinates and 3D point cloud coordinates is obtained by solving the parameters of the direct linear transformation equation based on the actual matching point pairs.

[0013] The coordinates of each point in the road image data are traversed, the corresponding pixel in the road point cloud data is found using the mapping relationship, the depth information of the pixel is assigned to the road image data, and a random sampling consistency algorithm is used to remove mismatches to obtain the point cloud projection image.

[0014] The point cloud projection image is assigned different color values ​​to the corresponding pixels in the point cloud projection image according to the Z-axis coordinate value of the laser point to generate a depth image. Labels are used to annotate the pixels of cracks and background in the depth image to construct a dataset.

[0015] The constructed dataset is divided into a training set and a validation set. The crack recognition model based on the four-dimensional information fusion convolutional neural network recognition algorithm is trained to obtain the trained crack recognition model. Point cloud data and image data of the road surface to be identified are collected and input into the trained crack recognition model to identify cracks. The crack recognition result of the road surface to be identified is output.

[0016] In one embodiment, the spatial transformation matrix is:

[0017]

[0018] The coordinates of a point in the world coordinate system are represented as (X... W Y W Z W The coordinates of a point in the camera coordinate system are represented as (X... C Y C Z C R is the rotation matrix, T is the translation matrix, and the point in the image coordinate system is represented as (X... p Yp The image coordinates of the origin of the pixel coordinate system are represented as (u0, v0), and the resolution of the pixel on the X-axis of the image coordinate system is represented as d. x The resolution of a pixel on the Y-axis of the image coordinate system is represented as d. y Points in the image coordinate system (X p y p The coordinates of ) in the pixel coordinate system are represented as (u, v), and M is the camera intrinsic parameter matrix.

[0019] In one embodiment, the grayscale distributions of the two-dimensional and three-dimensional images exhibit significant nonlinear differences, while the structures possess consistent characteristics. Feature extraction is performed on the projected image pairs to obtain local feature descriptors for the projected image pairs based on enhanced structural consistency, including:

[0020] The Sobel filter bank is used to extract multi-directional edge maps of the projected image pairs. Non-maximum suppression is applied to the multi-directional edge maps to preserve the edge structures with obvious features, resulting in sparse edge structure maps.

[0021] A method based on local contrast is used to extract the main common structural features in the projected image pairs and to establish a structure-guided image.

[0022] The structure-guided image is used to perform a guided filtering operation on the sparse edge structure map to enhance structural consistency, thereby obtaining a structural feature map.

[0023] The structural feature map is divided into sub-regions, and feature histograms of each sub-region are established. The feature histograms of each sub-region are combined to obtain local feature descriptors.

[0024] In one embodiment, the step of extracting feature points from the local feature descriptor, using nearest neighbor distance ratio detection as a matching strategy, and matching the feature points using the FAST feature point detector to obtain actual matching point pairs includes:

[0025] Feature points are extracted from the local feature descriptor. Nearest neighbor distance ratio detection is used as the matching strategy. The FAST feature point detector is used to match the feature points to obtain preliminary matching point pairs.

[0026] Based on the actual matching point pairs calculated from the reference matrix, the standard error between each preliminary matching point pair and the corresponding actual matching point pair is analyzed to determine the accuracy of each preliminary matching point pair.

[0027] The actual matching pairs are determined based on the accuracy of each preliminary matching pair.

[0028] In one embodiment, the expression for the nearest neighbor distance ratio is:

[0029] D(a0,b0)<η·D(a0,b1)

[0030] Where D(·,·) represents the Euclidean distance, b0 represents the feature vector in the current image that corresponds to the feature vector a0 in the reference image, b1 represents the feature vector in the current image that is the second closest to a0, and η is the ratio threshold in the nearest neighbor distance ratio.

[0031] In one embodiment, obtaining the mapping relationship between RGB image pixel coordinates and 3D point cloud coordinates by solving the parameters of the direct linear transformation equation based on the actual matching point pairs includes:

[0032] The parameters of the direct linear transformation equation are solved based on the actual matching point pairs to obtain the index relationship between the 3D point cloud and the 2D image.

[0033] Based on the index relationship between the 3D point cloud and the 2D image, the mapping relationship between the pixel coordinates of the RGB image and the coordinates of the 3D point cloud is obtained.

[0034] In one embodiment, obtaining the index relationship between the 3D point cloud and the 2D image by solving the parameters of the direct linear transformation equation based on the actual matched point pairs includes:

[0035] Assume the actual matching point A has coordinates (u) in the pixel coordinate system. A v A ), whose coordinates in the world coordinate system are (X A Y A Z A ), the resolution d of the pixel on the X and Y axes of the image coordinate system. x d y The image coordinates of the origin of the pixel coordinate system are represented as (u0, v0). The relationship between the image and the point cloud coordinate system is established as follows:

[0036]

[0037] Where R is the rotation matrix, T is the translation matrix, and l1~l 12 These are the parameters for direct linear transformation.

[0038] Let l 12 =1, and based on the relationship between the image and the point cloud coordinate system, the direct linear transformation equation is:

[0039]

[0040] l1~l 11 Substituting the pixel coordinates and world coordinates of all actual matched points as unknowns, we can formulate the following equation:

[0041]

[0042] Among them, (X) 1...n Y 1...n Z 1...n (u) represents the coordinates of matching points 1...n in the world coordinate system. 1...n v 1...n ) represents the coordinates of matching points 1...n in the pixel coordinate system.

[0043] Further transformation into normal equations:

[0044] L = (B T B) -1 B T W

[0045] In the formula: L=(l1,l2,l3,l4,l5,l6,l7,l8,l9,l 10 ,l 11 ) T

[0046] W = (-u1,-v1,-u2,-v2,…,-u n ,-v n ) T

[0047]

[0048] The superscript T indicates transpose.

[0049] Based on the stated normal equations, the linear transformation parameters l1~l can be solved. 11 This allows us to obtain the index relationship between the 3D point cloud and the 2D image.

[0050] The aforementioned method for identifying road cracks based on multi-level registration mapping of point cloud-RGB heterogeneous images acquires road point cloud data and road image data using a lidar and camera mounted on a drone. The point cloud data includes point coordinates and time information, while the image data includes time and size information. A frame-synchronous acquisition method is used to synchronize the point cloud and image data temporally, and a spatial transformation matrix is ​​used for spatial synchronization. This creates mutually matching projection image pairs. Based on the significant nonlinear differences in grayscale distribution but consistent structure between 2D and 3D images, feature extraction is performed on the projection image pairs to obtain local feature descriptors based on enhanced structural consistency. Feature points are extracted from these descriptors, and nearest neighbor distance ratio detection is used as the matching strategy. A FAST feature point detector is used to match the feature points, obtaining actual matching point pairs. The parameters of the direct linear transformation equation are then solved based on these matching point pairs to obtain the RGB image pixels. The mapping relationship between coordinates and 3D point cloud coordinates is established. The coordinates of each point in the road surface image data are traversed, and the corresponding pixels in the road surface point cloud data are found using the mapping relationship. The depth information of the pixels is assigned to the road surface image data, and a random sampling consistency algorithm is used to eliminate mismatches, obtaining a point cloud projection image to achieve accurate registration of heterogeneous images. The point cloud projection image is then assigned different color values ​​to corresponding pixels in the point cloud projection image according to the Z-axis coordinates of the laser points, generating a depth image. Labels are used to annotate the pixels of cracks and background in the depth image, constructing a dataset. The constructed dataset is divided into training and validation sets to achieve complementarity and enhancement of road surface crack information. A crack recognition model based on a four-dimensional information fusion convolutional neural network recognition algorithm is trained to obtain a well-trained crack recognition model with higher recognition efficiency and accuracy. Point cloud data and image data of the road surface to be identified are collected, input into the trained crack recognition model for crack recognition, and the crack recognition results of the road surface to be identified are output, thereby improving the efficiency and accuracy of automatic crack recognition. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating a method for identifying road surface cracks based on multi-level registration mapping of point cloud-RGB heterogeneous images in one embodiment.

[0052] Figure 2 This is a schematic diagram of the network structure of a crack recognition model based on a convolutional neural network recognition algorithm with four-dimensional information fusion in one embodiment;

[0053] Figure 3 This is a diagram of the drone equipment assembly platform provided in the embodiments of this application;

[0054] Figure 4This is a schematic diagram of the coarse registration method for time synchronization of road image point clouds provided in this application;

[0055] Figure 5 This is a schematic diagram illustrating the method for establishing multi-directional histograms based on structural feature maps provided in this application.

[0056] Figure 6 This is a schematic diagram of the actual matching points with the same name extracted based on the nearest neighbor distance ratio detection provided in this application;

[0057] Figure 7 This is a schematic diagram of the annotation of the road surface image dataset after multi-level registration provided in this application;

[0058] Figure 8 This is a schematic diagram of the model prediction results provided in this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] In one embodiment, such as Figure 1 As shown, a method for identifying road surface cracks based on multi-level registration mapping of point cloud-RGB heterogeneous images is provided, including the following steps:

[0061] Step S1: Use a lidar and camera mounted on a drone to collect road surface point cloud data and road surface image data. The road surface point cloud data includes point coordinate information and time information, and the road surface image data includes time information and size information.

[0062] Among them, road surface point cloud data can be point cloud data collected from the road surface.

[0063] Among them, road surface image data can be image data collected from the road surface.

[0064] This process involves designing the assembly methods for different sensor devices, calculating UAV flight parameters, and developing a data acquisition program to obtain road surface point cloud data and road surface image data. The road surface point cloud data includes point coordinate information and time information, while the road surface image data includes time information and size information. Through intrinsic and extrinsic parameter calibration, the extrinsic parameter matrix between the radar and the camera, as well as the camera's intrinsic parameter information, are calculated.

[0065] The overall equipment for collecting road surface point cloud data and road surface image data consists of five parts: a drone-mounted platform, a lidar system, a camera, a microcontroller unit (MCU), and an external power supply. The drone-mounted platform uses a hexagonal drone, employing a flight control module and a GPS+IMU inertial navigation system to acquire the drone's flight trajectory and attitude. The lidar uses a solid-state lidar system, which uses internal micro-vibrating mirrors (MEMS) to deflect five laser lines at 25-degree angles, scanning within a 120° horizontal and 25° vertical scanning range, receiving up to 750,000 reflection points per second. The camera is a full-frame camera with a resolution of 40 megapixels. The microcontroller unit uses a Raspberry Pi microcontroller, connecting to the data transmission interfaces of both the flight control module and the lidar.

[0066] In terms of mounting, the lidar and camera are fixed by a customized carbon fiber board frame and then hoisted onto the shock absorber of the drone platform. Due to their small size, the microcontroller and external power supply are directly bound to the shock absorber. To ensure the drone's endurance, an external power supply is also provided to power the lidar and microcontroller.

[0067] The flight parameters for the UAV are as follows: Flight altitude is the height of the UAV relative to the road surface, i.e., relative flight altitude, which does not change with the longitudinal slope of the road. The flight altitude H of the UAV is calculated using the convex lens imaging principle as follows:

[0068] a = N / L

[0069] H = f * GSD / a

[0070] In the formula, N is the single-side length of the camera sensor (CMOS), L is the single-side length of the image captured from the flight path, a is the pixel size, GSD is the ground size corresponding to a single pixel in the digital image, and f is the lens focal length.

[0071] Since it is necessary to study the image acquisition method for the entire road surface, the flight speed V is calculated using the forward overlap:

[0072] V = t × L(1 - O)

[0073] In the formula, t is the shooting time interval ( / s), that is, the frequency of taking a picture; O is the heading overlap, that is, the overlap between two adjacent pictures on the same heading, which is generally taken as 75%.

[0074] The drone's data acquisition program consists of: a flight control and flight control data reading program written using the DJI Onboard SDK, which can also receive and store radar point cloud data during drone flight; and a data acquisition program designed in C++ to record and synchronously control GPS, IMU, LiDAR, and camera.

[0075] Step S2: The road surface point cloud data and road surface image data are time-synchronized using the frame synchronous acquisition method, and the road surface point cloud data and road surface image data are spatially synchronized using the spatial transformation matrix to create mutually matching projection image pairs.

[0076] Time synchronization involves generating unique timestamps for each frame of point cloud and image data based on a high-precision GNSS clock signal. An industrial control computer provides a common time reference. The camera uses triggered exposure, recording the camera trigger time and correcting for data transmission delays to obtain the image's true timestamp. The LiDAR performs hardware time synchronization via Precise Time Protocol (PTP). During synchronization, the lowest-frequency LiDAR data is used to find the image data with the smallest time difference, achieving millisecond-level synchronization of point cloud and image data.

[0077] Spatial synchronization can be based on the principle of camera imaging to transform spatial coordinates, which involves the transformation between the world coordinate system, camera coordinate system, image coordinate system and pixel coordinate system, as well as three mathematical relationships: rotation, translation and mapping.

[0078] In one embodiment, the spatial transformation matrix is:

[0079]

[0080] The coordinates of a point in the world coordinate system are represented as (X... W Y W Z W The coordinates of a point in the camera coordinate system are represented as (X... C Y C Z C R is the rotation matrix, T is the translation matrix, and the point in the image coordinate system is represented as (X... p Y p The image coordinates of the origin of the pixel coordinate system are represented as (u0, v0), and the resolution of the pixel on the X-axis of the image coordinate system is represented as d. x The resolution of a pixel on the Y-axis of the image coordinate system is represented as d. y Points in the image coordinate system (X p Y p The coordinates of ) in the pixel coordinate system are represented as (u, v), and M is the camera intrinsic parameter matrix.

[0081] In one embodiment, the world coordinate system to camera coordinate system is:

[0082] Assuming the coordinates of a point in the world coordinate system (X... W Y W Z W ), the coordinates of a point in the camera coordinate system (X C YC Z C The transformation relationship between the two can be expressed as a homogeneous matrix by the rotation matrix R and the translation matrix T:

[0083]

[0084] The camera coordinate system to image coordinate system is:

[0085] Assuming the point (X) in the camera coordinate system C Y C Z C The projection transformation yields the point (X) in the image coordinate system. p Y p The distance from the center of the lens to the imaging plane is the lens focal length f. Ignoring distortion, by the principle of similar triangles, the transformation relationship between the two coordinate systems is:

[0086]

[0087] The image coordinate system to pixel coordinate system is:

[0088] Assuming the image coordinates of the origin of the pixel coordinate system are (u0, v0), and the pixel's resolution along the X and Y axes of the image coordinate system is d... x d y Then the point (X) in the image coordinate system p Y p The coordinates in the pixel coordinate system are:

[0089]

[0090] Where (u, v) represents a point (X) in the image coordinate system. p Y p The coordinates of the pixel coordinate system.

[0091] The camera intrinsic parameter matrix is:

[0092] With current sensor technology, it is entirely possible to assume that the distortion parameter is 0. Multiplying the matrices from the camera coordinate system to the image coordinate system and from the image coordinate system to the pixel coordinate system yields the camera intrinsic parameter matrix:

[0093]

[0094] The projection of the point cloud onto the image is:

[0095] Combining the transformation processes between the world coordinate system and the camera coordinate system, the camera coordinate system and the image coordinate system, the image coordinate system and the pixel coordinate system, and the camera intrinsic parameter matrix, we obtain the process of projecting the point cloud onto the image, which is the world coordinate (X) transformation. W Y W ZW → Camera coordinates (X) C Y C Z C → Image coordinates (X) p Y p → pixel coordinates (u, v), the above sequence can be represented by multiplying the next step on the left to obtain the spatial transformation matrix:

[0096]

[0097] Step S3: Based on the significant nonlinear differences in grayscale distribution between two-dimensional and three-dimensional images, while the structures exhibit consistent characteristics, feature extraction is performed on the projected image pairs to obtain local feature descriptors for the projected image pairs based on enhanced structural consistency.

[0098] In one embodiment, given the significant nonlinear differences in grayscale distribution between two-dimensional and three-dimensional images, while the structures exhibit consistent characteristics, feature extraction is performed on the projected image pairs to obtain local feature descriptors for the projected image pairs based on enhanced structural consistency, including:

[0099] The Sobel filter bank is used to extract multi-directional edge maps of the projected image pairs. Non-maximum suppression is applied to the multi-directional edge maps to preserve edge structures with obvious features, resulting in sparse edge structure maps. The main common structural features in the projected image pairs are extracted using a local contrast-based method to establish a structure guide image. The structure guide image is used to perform guided filtering on the sparse edge structure maps to enhance structural consistency, resulting in a structure feature map. The structure feature map is divided into sub-regions, and feature histograms of each sub-region are established. The feature histograms of each sub-region are combined to obtain a local feature descriptor.

[0100] Step S4: Extract feature points from the local feature descriptor, use nearest neighbor distance ratio detection as the matching strategy, and use the FAST feature point detector to match the feature points to obtain actual matching point pairs.

[0101] Among them, the FAST feature point detector can be the FAST (Features from accelerated segmenttest) feature point detection operator.

[0102] In one embodiment, feature points are extracted from the local feature descriptor, and the nearest neighbor distance ratio detection is used as the matching strategy. The FAST feature point detector is used to match the feature points to obtain actual matching point pairs, including:

[0103] Feature points are extracted from local feature descriptors. Nearest neighbor distance ratio detection is used as the matching strategy. The FAST feature point detector is used to match the feature points to obtain preliminary matching point pairs. The true matching point pairs are calculated based on the reference matrix. The standard error between each preliminary matching point pair and the corresponding true matching point pair is analyzed to determine the accuracy of each preliminary matching point pair. Based on the accuracy of each preliminary matching point pair, the actual matching point pairs are determined.

[0104] Among them, the nearest neighbor distance ratio (NNDR) is used as an optimization strategy for feature matching. The NNDR strategy can be expressed as follows:

[0105] D(a0,b0)<η·D(a0,b1)

[0106] In the formula, D(·,·) represents the Euclidean distance, b0 represents the feature vector in the current image that corresponds to the feature vector a0 in the reference image, b1 represents the feature vector in the current image that is the second closest to a0, and η is the proportional threshold in the nearest neighbor distance ratio.

[0107] The reference image can be a standard image with reference calibration points on the lidar and camera.

[0108] Step S5: Solve the parameters of the direct linear transformation equation based on the actual matching point pairs to obtain the mapping relationship between the pixel coordinates of the RGB image and the coordinates of the three-dimensional point cloud.

[0109] In one embodiment, the mapping relationship between RGB image pixel coordinates and 3D point cloud coordinates is obtained by solving the parameters of the direct linear transformation equation based on the actual matched point pairs, including:

[0110] Based on the parameters of the direct linear transformation equation obtained by solving the actual matching point pairs, the index relationship between the 3D point cloud and the 2D image is obtained; based on the index relationship between the 3D point cloud and the 2D image, the mapping relationship between the pixel coordinates of the RGB image and the coordinates of the 3D point cloud is obtained.

[0111] In one embodiment, the index relationship between the 3D point cloud and the 2D image is obtained by solving the parameters of the direct linear transformation equation based on the actual matched point pairs, including:

[0112] Assume the actual matching point A has coordinates (u) in the pixel coordinate system. A v A ), whose coordinates in the world coordinate system are (X A Y A Z A ), the resolution d of the pixel on the X and Y axes of the image coordinate system. x d yThe image coordinates of the origin of the pixel coordinate system are represented as (u0, v0). The relationship between the image and the point cloud coordinate system is established as follows:

[0113]

[0114] Where R is the rotation matrix, T is the translation matrix, and l1~l 12 These are the parameters for direct linear transformation.

[0115] Let l 12 =1, and based on the relationship between the image and the point cloud coordinate system, the direct linear transformation equation is:

[0116]

[0117] l1~l 11 Substituting the pixel coordinates and world coordinates of all actual matched points as unknowns, we can formulate the following equation:

[0118]

[0119] Among them, (X) 1...n Y 1...n Z 1...0n (u) represents the coordinates of matching points 1...n in the world coordinate system. 1...n v 1...n ) represents the coordinates of matching points 1...n in the pixel coordinate system.

[0120] Further transformation into normal equations:

[0121] L = (B T B) -1 B T W

[0122] In the formula: L=(l1,l2,l3,l4,l5,l6,l7,l8,l9,l 10 ,l 11 ) T

[0123] W = (-u1,-v1,-u2,-v2,…,-u n ,-v n ) T

[0124]

[0125] The superscript T indicates transpose.

[0126] Based on the stated normal equations, the linear transformation parameters l1~l can be solved. 11 This allows us to obtain the index relationship between the 3D point cloud and the 2D image.

[0127] Step S6: Traverse the coordinates of each point in the road surface image data, use the mapping relationship to find the corresponding pixel in the road surface point cloud data, assign the depth information of the pixel to the road surface image data, and use the random sampling consistency algorithm to remove mismatches to obtain the point cloud projection image.

[0128] It should be understood that after obtaining the index relationship between the 3D point cloud and the 2D image, the mapping relationship between the RGB image pixel coordinates and the 3D point cloud coordinates is obtained. By traversing the coordinates of each point in the road image data, the corresponding pixel in the road point cloud data is found using the mapping relationship. The depth information of the road point cloud data can then be assigned to the road image data. The Random Sample Consensus (RANSAC) algorithm is used to iteratively reduce errors, thereby achieving precise registration between the image and the point cloud.

[0129] Step S7: Assign different color values ​​to the corresponding pixels in the point cloud projection image according to the Z-axis coordinates of the laser points to generate a depth image. Use Labelme to annotate the pixels of the crack and background in the depth image to construct a dataset.

[0130] Step S8: Divide the constructed dataset into a training set and a validation set, train the crack recognition model based on the four-dimensional information fusion convolutional neural network recognition algorithm to obtain the trained crack recognition model, collect point cloud data and image data of the road surface to be identified, input them into the trained crack recognition model to perform crack recognition, and output the crack recognition result of the road surface to be identified.

[0131] The crack recognition model performs convolutional layer fusion and fully connected layer fusion on point cloud data and image data to obtain the fused feature vector AFeature and the fused feature BFeature. The fused feature vector AFeature and the fused feature BFeature are then fused to obtain the feature FC Fusion. Crack recognition is performed based on the feature FC Fusion, and the crack recognition result is output.

[0132] The network structure of the crack recognition model is as follows: Figure 2As shown, the network structure includes three improved fusion structures, FCNN. FCNN-RGBD takes 4-channel RGB-D information as input, and the color and depth images are fused in the first convolutional layer, outputting a feature vector of 256 neurons. This feature vector, AFeature, is obtained by dimensionality transformation and feature integration through a fully connected layer with 512 neurons. FCNN-RGB takes a color image as input and outputs a color feature vector of 256 neurons. FCNN-Depth takes a depth image as input and outputs a depth feature vector of 256 neurons. The two feature vectors are fused through a fully connected layer to obtain a fused feature vector, BFeature. Finally, the two fused feature vectors, AFeature and BFeature, are fused again through a fully connected layer to obtain the feature FC Fusion. Crack recognition is performed based on the feature FC Fusion, and the crack recognition result is output, achieving multi-dimensional information fusion recognition.

[0133] The aforementioned method for identifying road cracks based on multi-level registration mapping of point cloud-RGB heterogeneous images acquires road point cloud data and road image data using a lidar and camera mounted on a drone. The point cloud data includes point coordinates and time information, while the image data includes time and size information. A frame-synchronous acquisition method is used to synchronize the point cloud and image data temporally, and a spatial transformation matrix is ​​used for spatial synchronization. This creates mutually matching projection image pairs. Based on the significant nonlinear differences in grayscale distribution but consistent structure between 2D and 3D images, feature extraction is performed on the projection image pairs to obtain local feature descriptors based on enhanced structural consistency. Feature points are extracted from these descriptors, and nearest neighbor distance ratio detection is used as the matching strategy. A FAST feature point detector is used to match the feature points, obtaining actual matching point pairs. The parameters of the direct linear transformation equation are then solved based on these matching point pairs to obtain the RGB image pixels. The mapping relationship between coordinates and 3D point cloud coordinates is established. The coordinates of each point in the road surface image data are traversed, and the corresponding pixels in the road surface point cloud data are found using the mapping relationship. The depth information of the pixels is assigned to the road surface image data, and a random sampling consistency algorithm is used to eliminate mismatches, obtaining a point cloud projection image to achieve accurate registration of heterogeneous images. The point cloud projection image is then assigned different color values ​​to corresponding pixels in the point cloud projection image according to the Z-axis coordinates of the laser points, generating a depth image. Labels are used to annotate the pixels of cracks and background in the depth image, constructing a dataset. The constructed dataset is divided into training and validation sets to achieve complementarity and enhancement of road surface crack information. A crack recognition model based on a four-dimensional information fusion convolutional neural network recognition algorithm is trained to obtain a well-trained crack recognition model with higher recognition efficiency and accuracy. Point cloud data and image data of the road surface to be identified are collected, input into the trained crack recognition model for crack recognition, and the crack recognition results of the road surface to be identified are output, thereby improving the efficiency and accuracy of automatic crack recognition.

[0134] In one embodiment, the specific steps of a road surface crack identification method based on multi-level registration mapping of point cloud-RGB heterogeneous images are as follows:

[0135] Step 1: Design the assembly method, flight parameters and acquisition program of each sensor, and obtain matching point cloud and image data, including time information, point cloud coordinate system, camera coordinate system, calibration extrinsic information and intrinsic information;

[0136] First, a multi-source detection system combining lidar and camera is designed based on a UAV mobile platform. The lidar, camera, and mobile platform are jointly calibrated to determine the relative position and attitude of each sensor, and to collect road point cloud data and road image data.

[0137] The drone platform used is the DJI Matrice 600 Pro, a six-rotor flight platform designed for industrial applications. It has excellent load-bearing capacity, with a maximum takeoff weight of 15.5kg. It is also equipped with a customized DJ A3 Pro triple-redundant flight control system and a Lighten Bridge2 image transmission module. It not only has rich communication and SDK interfaces to meet the customized needs of professional applications, but also can transmit images to the remote control terminal in real time and control the camera shutter.

[0138] The camera used is the Sony A7R3 mirrorless camera, with a 36mm x 24mm full-frame image sensor and a resolution exceeding 40 megapixels. It is equipped with an FE 2.8 / 35mm fixed-focus lens. For control, the camera is integrated with the flight control system and features two sync interfaces via DJI's developer toolkit for shutter triggering, allowing for one-button control of the camera from the remote control. For mounting, a custom-made carbon fiber frame was used. The camera was first secured to the frame, keeping its optical axis as parallel as possible, and then the frame was mounted on a shock-absorbing mount beneath the drone platform.

[0139] The lidar is Robosense's RS-Lidar-M1 solid-state lidar. It uses internal micro-vibrating mirrors (MEMS) to deflect five laser lines at 25 angles, achieving the same scanning effect as a traditional 128-line mechanical lidar within a scanning range of 120° horizontally and 25° vertically. It can receive up to 750,000 reflection points per second, with an average angular resolution of 0.2° and a maximum ranging distance of up to 200m.

[0140] The microcontroller selected is a Raspberry Pi 4b microcontroller, which is connected to the data transmission interfaces of the A3 Pro flight control module and the LiDAR.

[0141] To ensure the drone's endurance, an external power supply is mounted on the carbon fiber plate to power the LiDAR and microcontroller. The specific assembly method is as follows: Figure 3 As shown.

[0142] By developing flight control and data transmission programs using the DJ Onboard SDK from the DJI Developer Toolkit, it is possible to simultaneously control the drone, the shooting of various sensors, and the data transmission.

[0143] Step 2: Perform time and spatial synchronization to achieve coarse registration of point cloud images;

[0144] Time synchronization: For different sensor devices, a data acquisition program based on C++ language was developed to synchronously record the start and end times of acquisition from GPS, IMU, LiDAR, and camera on the industrial control console, thereby unifying the time reference system and realizing real-time correspondence between road point cloud data and road image data.

[0145] In this embodiment, the high-definition camera operates at 30 FPS, and the lidar operates at 15 FPS. The significant difference in operating frequencies between these two heterogeneous sensors results in an inherent time discrepancy between the road point cloud data and the road image data. The combined inertial navigation system on the UAV mobile platform provides latitude and longitude coordinates and a high-precision clock signal at a frequency of 100 Hz. Here, a timestamp alignment method based on the clock signal is used to achieve millisecond-level synchronization between the road point cloud data and the road image data. Figure 4 As shown.

[0146] Specifically, a timestamp is generated for each frame of road point cloud data and road image data based on a high-precision GNSS clock signal. During synchronization, the image data with the smallest time difference is found using low-frequency LiDAR data. In this embodiment, taking a 15Hz LiDAR and a 30Hz camera as an example, the maximum time deviation is less than 10ms, and the UAV travels at a speed of 20m / s with a distance deviation of 0.2m, which basically meets the accuracy requirements for data acquisition.

[0147] Spatial synchronization: Spatial synchronization involves establishing a camera imaging model and performing distortion correction. Establishing the camera model enables the transformation of the target from 3D world coordinates to pixel coordinates, while distortion correction primarily aims to correct aberrations. The required parameters include: point cloud, rotation matrix from point cloud coordinates to camera coordinates, translation vector from point cloud coordinates to camera coordinates, camera intrinsic parameters, and camera distortion parameters.

[0148] After spatial projection transformation, the camera sensor position and orientation are specified, and an appropriate projection plane is set. Based on these parameters, a coarsely registered projected image pair can be created.

[0149] Step 3: Construct local feature descriptors:

[0150] After obtaining the coarsely registered projected image pairs, feature point extraction is required.

[0151] This embodiment establishes a directional structure histogram and local feature descriptors based on enhanced structural consistency.

[0152] First, an edge structure map of the image is established, and an extended Sobel filter is used to extract multi-directional structural information from the two-dimensional and three-dimensional images. In this embodiment, the filter is extended and improved; compared to the traditional bidirectional Sobel filter, the extended Sobel filter has five directions.

[0153] Let the input image be I(x,y), f n (x, y), n = 1, 2, 3, 4, 5 represent the Sobel filter bank, with numbers from 1 to 5 representing (a) 0°, (b) 45°, (c) 90°, (d) 135°, and (e) no direction, respectively. The multi-directional Sobel edge structure map extraction can then be represented as:

[0154] SE n (x,y)=|I(x,y)*f n (x,y)|

[0155] In the formula, SE n (x,y) represents the Sobel edge at pixel (x,y) along direction n, "*" indicates convolution, and "|·|" indicates taking the absolute value.

[0156] Next, SE n The grayscale range of (x,y) is stretched to (0, 255), and non-maximum suppression is performed. At each position, the values ​​are compared point by point along the direction, and the maximum value at each point on the edge of the direction is retained, resulting in an edge structure map EM for 5 directions. n .

[0157] Secondly, based on local contrast extraction of the guide image, the mean of all gray levels within the local region is calculated, and the absolute value of the difference between this mean and the gray level of each pixel within the region is taken. Then, all absolute values ​​are normalized to a value between (0, 1) and accumulated to obtain the gray level of the guide image G. This process can be represented as follows:

[0158]

[0159] In the formula, I is the source image, G is the guide image, and (x, y) represents the center of the local window. mean This represents the average grayscale value of all pixels within a local area, where i and j represent the changes in pixel values ​​within the local area, and m represents the maximum value of pixel changes within the local area.

[0160] Next, a structural feature map is established based on structural consistency enhancement, using the guide image G and edge structure map EM constructed above. n Structural consistency enhancement is achieved through guided filtering, and a structural feature map (SM) is established. n Guided filtering uses a guide graph to constrain the filtering process of the input image. The output image can be regarded as the result of a linear transformation of the guide graph within a local window. The linear coefficients are solved by minimizing the difference between the output image and the input image.

[0161] Finally, a method for generating local feature descriptors based on multi-directional histograms derived from structural feature maps is used. This involves sub-regional division, where a 32×32 pixel region centered on a feature point is selected on the feature map and divided into 16 4×4 sub-regions. The pixel values ​​in each direction of each 8×8 sub-region are statistically analyzed and arranged sequentially to create a 5-directional feature histogram. The histograms of each region are normalized using the L2 norm, and the 16 histograms are combined to obtain an 80-dimensional feature descriptor. The road surface image segmentation process in this embodiment is as follows: Figure 5 As shown.

[0162] Step 4: Solve the parameters of the direct linear transformation equation to achieve precise registration of the point cloud image;

[0163] Feature points are extracted using the local feature descriptor constructed in step 3. Then, the FAST feature point detector is used, and nearest neighbor distance ratio detection is used as the matching strategy to perform feature point matching, resulting in actual matching point pairs. In this embodiment, the above operation yielded 22 pairs of actual matching points with the same name. The actual matching points with the same name are as follows: Figure 6 As shown, the coordinates of the actual matching points with the same name are substituted into the direct linear transformation equation to calculate the direct linear transformation parameters, as shown in Table 1:

[0164] coefficient value coefficient value <![CDATA[l1]]> -40.248910 <![CDATA[l7]]> 70.475391 <![CDATA[l2]]> -40.515524 <![CDATA[l8]]> -245.073019 <![CDATA[l3]]> -8.384885 <![CDATA[l9]]> -0.022806 <![CDATA[l4]]> -326.330583 <![CDATA[l 10 ]]> -0.046011 <![CDATA[l5]]> 13.240888 <![CDATA[l 11 ]]> 0.020568 <![CDATA[l6]]> 5.047637

[0165] After obtaining the direct linear transformation parameters, the correspondence between the 3D point cloud and the 2D image is obtained, allowing for data fusion. Since the point cloud depth image and the depth image are highly homogeneous, they can share the mapping matrix. Using the mapping matrix, the Random Sample Consensus (RANSAC) algorithm is used to iteratively reduce errors, achieving precise registration between the image and the point cloud.

[0166] Step 5: Crack information annotation and dataset creation;

[0167] Based on the above registration method, a dataset of RGB road surface images and point cloud depth images corresponding to spatial coordinates was established. Multiple edge filters were used for feature extraction to analyze the crack edge features in heterogeneous images. Labels were used to annotate the images pixel by pixel, labeling them as cracks and background. The final dataset is shown below. Figure 7 As shown.

[0168] Step 6: Train and validate a crack recognition model based on a convolutional neural network algorithm using four-dimensional information fusion;

[0169] Using the created dataset, a crack recognition model based on a four-dimensional information fusion convolutional neural network algorithm was designed by fusing depth information from images. The network structure is as follows: Figure 2 As shown.

[0170] The network contains three improved fusion structures, FCNN. FCNN-RGBD takes 4-channel RGB-D information as input. The color image and depth image are fused in the first convolutional layer, and the output is a feature vector composed of 256 neurons. The fused feature vector AFeature is obtained by dimensional transformation and feature integration through a fully connected layer with 512 neurons.

[0171] FCNN-RGB takes a color image as input and outputs a color feature vector composed of 256 neurons; FCNN-Depth takes a depth image as input and outputs a depth feature vector composed of 256 neurons. The two feature vectors are fused through a fully connected layer to obtain the fused feature vector BFeature.

[0172] Finally, the two fused feature vectors, AFeature and BFeature, are fused again through a fully connected layer to achieve multi-dimensional information fusion recognition.

[0173] During experimental training, Stochastic Gradient Descent (SGD) was used as the optimizer, and Mean Absolute Error (MAE) was used as the loss function. A training cycle of 300 epochs was set, and the dataset was divided into training and validation sets in a 7:3 ratio to train the crack recognition model based on a convolutional neural network algorithm using four-dimensional information fusion. Using the trained weight file, the bounding box, classification name, and probability of the target can be detected and output, completing the road crack recognition based on a drone. Figure 8 As shown.

[0174] The identification results include the category, confidence level, and coordinates of the predicted bounding box for each crack. Simultaneously, the actual mileage marker can be calculated using GPS information acquired during image acquisition, thus outputting the road crack classification and marker location in text form. This completes the identification and recording of road cracks, providing a basis for road maintenance management.

[0175] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0176] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for identifying road surface cracks based on multi-level registration and mapping of point cloud-RGB heterogeneous images, characterized in that, The method includes: The road surface point cloud data and road surface image data are collected using a lidar and camera mounted on a drone. The road surface point cloud data includes point coordinate information and time information, and the road surface image data includes time information and size information. The road surface point cloud data and the road surface image data are time-synchronized using a frame-synchronized acquisition method, and the road surface point cloud data and the road surface image data are spatially synchronized using a spatial transformation matrix to create mutually matching projection image pairs. Based on the significant nonlinear differences in grayscale distribution between two-dimensional and three-dimensional images, while maintaining structural consistency, feature extraction is performed on the projected image pairs to obtain local feature descriptors for the projected image pairs based on enhanced structural consistency. Feature points are extracted from the local feature descriptor. Nearest neighbor distance ratio detection is used as the matching strategy. The FAST feature point detector is used to match the feature points to obtain actual matching point pairs. The mapping relationship between RGB image pixel coordinates and 3D point cloud coordinates is obtained by solving the parameters of the direct linear transformation equation based on the actual matching point pairs. The coordinates of each point in the road image data are traversed, the corresponding pixel in the road point cloud data is found using the mapping relationship, the depth information of the pixel is assigned to the road image data, and a random sampling consistency algorithm is used to remove mismatches to obtain the point cloud projection image. The point cloud projection image is assigned different color values ​​to the corresponding pixels in the point cloud projection image according to the Z-axis coordinate value of the laser point to generate a depth image. Labels are used to annotate the pixels of cracks and background in the depth image to construct a dataset. The constructed dataset is divided into a training set and a validation set. The crack recognition model based on the convolutional neural network recognition algorithm of four-dimensional information fusion is trained to obtain the trained crack recognition model. Point cloud data and image data of the road surface to be identified are collected and input into the trained crack recognition model to identify cracks. The crack recognition result of the road surface to be identified is output. The grayscale distributions of the two-dimensional and three-dimensional images exhibit significant nonlinear differences, while the structures share a consistent characteristic. Feature extraction is performed on the projected image pairs to obtain local feature descriptors for the projected image pairs based on enhanced structural consistency, including: The Sobel filter bank is used to extract multi-directional edge maps of the projected image pairs. Non-maximum suppression is applied to the multi-directional edge maps to preserve distinctive edge structures, resulting in a sparse edge structure map. The Sobel filter bank has five directions. The extraction expression for the multi-directional edge map is as follows: ； in, Indicated in pixels The Sobel edge is located along direction n. "*" indicates convolution, "|·|" indicates taking the absolute value, and n=1, 2, 3, 4, 5 represent the Sobel filter bank. The numbers from 1 to 5 represent 0°, 45°, 90°, 135° and no direction, respectively. For one of the images in the projected image pair, Another image in a projected image pair; A method based on local contrast is used to extract the main common structural features in the projected image pairs, and a structural guidance image is established. The structural guidance image is as follows: ； Where I is the source image in the projected image pair, G is the structure-guided image, and (x, y) represents the center of the local window. This represents the average gray value of all pixels within a local area, where i and j represent the changes in pixel values ​​within the local area, and m represents the maximum value of pixel changes within the local area. The structure-guided image is used to perform a guided filtering operation on the sparse edge structure map to enhance structural consistency, thereby obtaining a structural feature map. For the structural feature map, take a 32×32 pixel region centered on the feature point and divide it into 16 4×4 sub-regions; count the pixel values ​​in each direction of each 8×8 sub-region and arrange them in order to build a 5-direction feature histogram; normalize the histogram of each region using the L2 norm, and combine the 16 histograms to obtain an 80-dimensional feature descriptor; The step of extracting feature points from the local feature descriptor, using nearest neighbor distance ratio detection as the matching strategy, and using the FAST feature point detector to match the feature points to obtain actual matching point pairs includes: Feature points are extracted from the local feature descriptor. Nearest neighbor distance ratio detection is used as the matching strategy. The FAST feature point detector is used to match the feature points to obtain preliminary matching point pairs. Based on the actual matching point pairs calculated from the reference matrix, the standard error between each preliminary matching point pair and the corresponding actual matching point pair is analyzed to determine the accuracy of each preliminary matching point pair. Based on the accuracy of each preliminary matching point pair, the actual matching point pairs are determined; The expression for the nearest neighbor distance ratio is: ； Wherein, D(· ·) represents Euclidean distance. This indicates the feature vector in the current image that corresponds to the feature vector in the reference image. The most recent feature vector Indicates the current image for The second nearest eigenvector It is the proportion threshold in the nearest neighbor distance ratio.

2. The method according to claim 1, characterized in that, The spatial transformation matrix is: ； The coordinates of a point in the world coordinate system are represented as ( , , The coordinates of a point in the camera coordinate system are represented as ( , , ), For rotation matrix, Given a translation matrix, the points in the image coordinate system are represented as ( , The image coordinates of the origin of the pixel coordinate system are represented as ( , The resolution of a pixel on the X-axis of the image coordinate system is expressed as... The resolution of a pixel on the Y-axis of the image coordinate system is expressed as Points in the image coordinate system ( , The coordinates of ) in the pixel coordinate system are represented as ( , ), This is the camera intrinsic parameter matrix.

3. The method according to claim 1, characterized in that, The step of solving the direct linear transformation equation based on the actual matching point pairs to obtain the mapping relationship between RGB image pixel coordinates and 3D point cloud coordinates includes: The parameters of the direct linear transformation equation are solved based on the actual matching point pairs to obtain the index relationship between the 3D point cloud and the 2D image. Based on the index relationship between the 3D point cloud and the 2D image, the mapping relationship between the pixel coordinates of the RGB image and the coordinates of the 3D point cloud is obtained.

4. The method according to claim 3, characterized in that, The step of obtaining the index relationship between the 3D point cloud and the 2D image by solving the parameters of the direct linear transformation equation based on the actual matching point pairs includes: Assume the coordinates of the actual matching point A in the pixel coordinate system are ( , The coordinates of ) in the world coordinate system are ( , , (Pixel resolution on the X and Y axes of the image coordinate system) , The image coordinates of the origin of the pixel coordinate system are represented as ( , The relationship between the image and the point cloud coordinate system is established as follows: ； in, For rotation matrix, It is a translation matrix. ~ These are the parameters for direct linear transformation; make =1, and based on the relationship between the image and the point cloud coordinate system, the direct linear transformation equation is: ； Will ~ Substituting the pixel coordinates and world coordinates of all actual matched points as unknowns, we can formulate the following equation: ； in,( , , () represents the coordinates of matching points 1...n in the world coordinate system. , () represents the coordinates of matching points 1...n in the pixel coordinate system; Further transformation into normal equations: ； In the formula: ; ； ； The superscript T indicates transpose; Based on the stated normal equations, the linear transformation parameters can be solved. ~ This allows us to obtain the index relationship between the 3D point cloud and the 2D image.

Images