A point cloud-image registration method, system, device and medium for bridge disease spatial form detection

Abstract

The application discloses a point cloud-image registration method for bridge disease spatial form detection, comprising the following steps: extracting a point cloud main plane normal vector, constructing a rotation matrix to make the point cloud main plane parallel to the XZ plane; projecting the rotated point cloud in the main plane direction into a gray scale image, and extracting the disease edge feature points and geometric edge feature points therein; searching for three-dimensional edge feature points corresponding to the extracted edge feature points in the original point cloud; generating a gray scale image for the target image, extracting the geometric edge features of the disease and structure, and constructing an image edge KD tree; projecting the point cloud three-dimensional edge feature points to the target image plane and performing nearest neighbor matching with the extracted image edge feature points; constructing a 6-DOF external parameter optimization function to optimize the rotation matrix and translation vector; and projecting the complete point cloud to the image plane. The application can realize high-precision three-dimensional point cloud and image data fusion, and improve the bridge disease recognition capability and detection precision.

Description

Technical Field

[0001] This invention belongs to the technical field of bridge safety assessment, and in particular relates to a point cloud-image registration method, system, equipment and medium for detecting the spatial morphology of bridge defects. Background Technology

[0002] In bridge maintenance and defect detection, accurately determining the location and three-dimensional morphology of bridge structural defects is crucial for guiding maintenance design and assessing structural condition. Commonly used image-based defect detection methods can only perform planar measurements of defects and rely on physical calibration of pixel dimensions, failing to support spatial measurement and severity assessment of bridge defects. While 3D point cloud technology can reflect the geometric features of the structure, it lacks texture information; registering and fusing point clouds with images is one of the key technologies for achieving multi-dimensional information complementarity and intelligent spatial morphology recognition.

[0003] However, current point cloud and image registration methods mostly rely on manual annotation, coarse registration, or are only applicable to regular structure scenarios, which makes it difficult to meet the actual needs of complex shapes, high precision, and automation in bridge defect detection. Summary of the Invention

[0004] Purpose of the invention: The first purpose of this invention is to provide a point cloud-image registration method for spatial morphology detection of bridge defects, which can achieve high-precision fusion of three-dimensional point cloud and image data, thereby improving the ability to identify and detect bridge defects.

[0005] The second objective of this invention is to provide a point cloud-image registration system for detecting the spatial morphology of bridge defects.

[0006] A third objective of this invention is to provide an electronic device.

[0007] A fourth objective of this invention is to provide a computer-readable storage medium.

[0008] Technical Solution: To achieve the above objectives, this invention discloses a point cloud-image registration method for spatial morphology detection of bridge defects, comprising the following steps:

[0009] S1. Extract the normal vector of the principal plane of the point cloud and construct a rotation matrix to make the principal plane of the point cloud parallel to the XZ plane;

[0010] S2. Based on the intensity value of the point cloud, project the rotated point cloud from step S1 onto the main plane as a grayscale image, and extract the disease edge feature points and geometric edge feature points; extract the three-dimensional edge feature points corresponding to the edge feature points in the original point cloud through projection coordinates and KD tree search.

[0011] S3. First, generate a grayscale image for the target image, then extract the geometric edge features of the lesions and structures, and construct an image edge KD tree;

[0012] S4. Combining the initial extrinsic parameters, camera intrinsic parameters, and distortion parameters, the 3D edge feature points of the point cloud extracted in step S2 are projected onto the target image plane and matched with the nearest neighbor of the image edge feature points extracted in step S3. With the average projection error as the objective, a 6-DOF extrinsic parameter optimization function is constructed to optimize the rotation matrix and translation vector.

[0013] S5. Using the optimized extrinsic parameters, the complete point cloud is projected onto the image plane to achieve final registration.

[0014] Optionally, step S2 specifically includes the following steps:

[0015] S201. For the point cloud after rotating the main plane to be parallel to the XZ plane in step S1, extract the three-dimensional coordinates (x, y, z) and intensity value In from the point cloud data P, and project the three-dimensional point cloud onto the XZ plane. Construct a KD tree based on the coordinates of the projected points. Then normalize the intensity value to the range of [0, 1], and the normalized intensity value is NIn. Finally, map the normalized intensity value to the gray value gv to generate a grayscale image of the point cloud.

[0016] S202. Based on the point cloud grayscale image generated in step S201, the Canny algorithm is used to extract the disease edge feature points and geometric edge feature points. The edge features extracted in the point cloud grayscale image are edge points GP. edg The set; for each edge point GP edg The corresponding projected coordinates are (u0, v0). A KD-tree is used to search for the nearest point in the original point cloud, and edge points GP are extracted from the original point cloud based on the KD-tree search results. edg The corresponding point cloud P edg Its three-dimensional coordinates are (x pe ,y pe ,z pe ).

[0017] Optionally, step S3 specifically includes the following steps:

[0018] S301. First, generate a grayscale image of the target image. Then, use the Canny algorithm to extract its disease edge feature points and geometric edge feature points. The edge features extracted from the point cloud grayscale image are edge points GI. edg The set of each edge feature point GI edg The coordinates in the image coordinate system are (u ie ,v ie );

[0019] S302, Based on the feature points GI extracted in step S301 edg The set of edge points is used to construct a KD-tree-based data structure.

[0020] Optionally, step S4 specifically includes the following steps:

[0021] S401. Assume the initial transformation matrix T from the point cloud coordinate system to the camera coordinate system. in T in From the rotation matrix R in Translation vector t in Composition, R in Given a 3×3 matrix, t in Given a 3×1 vector; based on the initial transformation matrix T in The extracted feature points P from the point cloud edg Transform to camera coordinate system to obtain P edgc The corresponding camera coordinates are (x c ,y c ,z c );

[0022] P edgc =R in P edg +t in (1)

[0023] S402, Based on the feature point P obtained in step S401 edgc and the corresponding camera coordinates (x) c ,y c ,z c Based on the camera intrinsic parameter matrix K and distortion coefficients, the feature points in the extracted point cloud are projected onto the image plane; based on the coordinates (x, y) in the camera coordinate system... c ,y c ,z c The normalized planar coordinates are obtained as follows:

[0024]

[0025] Feature point P after considering camera distortion edgc The normalized plane coordinates are (x dist ,y dist ):

[0026]

[0027] Where k1, k2, and k3 are the radial distortion coefficients of the camera; p1 and p2 are the tangential distortion coefficients of the camera.

[0028] Based on the camera's intrinsic parameter matrix K, the feature points P are...edgc Transform to the image plane to obtain feature point P edgi Its corresponding pixel coordinates on the image plane are (u pe ,v pe ),

[0029]

[0030] Wherein, the camera intrinsic parameter matrix K is a 3×3 matrix:

[0031]

[0032] Among them, c x c y f is the coordinate of the optical center on the pixel plane. x f y These are the focal lengths that represent the image scaling along the x-axis and y-axis, respectively;

[0033] Point cloud feature points P edgc Transform the feature point P from the camera coordinate system to the image plane coordinate system. edgi The process, i.e. the process from formula (2) to formula (7), is represented by the function π:

[0034] P edgi =π(P edgc (8)

[0035] According to formula (1), the feature points P of the point cloud edg Transform the edge feature point P from the point cloud coordinate system to the image plane coordinate system. edgi Represented as:

[0036] P edgi =π(R) in P edg +t in (9)

[0037] S403, Based on the edge feature points P of the point cloud projected onto the camera plane. edgi Coordinates (u) on the camera plane pe ,v pe ) and the image edge feature points GI extracted in step S301 edg Coordinates (u) on the camera plane ie ,v ie Perform nearest neighbor matching; for the j-th feature point P edgij The nearest edge point GI is found using the KD tree established in step S302. edgj And calculate the distance between them as d. j ;

[0038] d j =||Pedgij -GI edgj || (10)

[0039] After calculating the distances to all projected points, take all distances d. j The sum of the first 90% of the distances is used to construct the optimization objective function:

[0040]

[0041] Where m is the feature point P edgi The number of midpoints;

[0042] According to formula (9), the optimization objective function can be obtained:

[0043]

[0044] Where R is the rotation matrix to be optimized; t is the translation vector to be optimized;

[0045] By optimizing the rotation matrix R and the translation vector t to minimize the value of the objective function, the final rotation matrix R is obtained. ul Translation vector t ul .

[0046] Based on the same inventive concept, the point cloud-image registration system for spatial morphology detection of bridge defects described in this invention includes:

[0047] The point cloud processing alignment module is used to extract the normal vector of the principal plane of the point cloud and construct a rotation matrix to make the principal plane of the point cloud parallel to the XZ plane.

[0048] The feature point extraction module is used to project the point cloud rotated in the point cloud processing and alignment module into a grayscale image in the main plane direction based on the intensity value of the point cloud, and extract the disease edge feature points and geometric edge feature points; and extract the three-dimensional edge feature points corresponding to the edge feature points in the original point cloud through projection coordinates and KD tree search.

[0049] The image edge extraction module is used to first generate a grayscale image of the target image, and then extract the geometric edge features of the lesions and structures in it to construct an image edge KD tree;

[0050] The edge pairing module combines initial extrinsic parameters, camera intrinsic parameters, and distortion parameters to project the 3D edge feature points of the point cloud extracted by the feature point extraction module onto the target image plane and perform nearest neighbor matching with the image edge feature points extracted by the image edge extraction module; with the average projection error as the objective, a 6-DOF extrinsic parameter optimization function is constructed to optimize the rotation matrix and translation vector;

[0051] The projection registration module is used to project the complete point cloud onto the image plane using optimized extrinsic parameters to achieve final registration.

[0052] Optionally, in the feature point extraction module, after the point cloud processing alignment module rotates the main plane to be parallel to the XZ plane, the three-dimensional coordinates (x, y, z) and intensity value In of the point cloud data P are extracted, and the three-dimensional point cloud is projected onto the XZ plane. A KD tree is constructed based on the projection point coordinates. Then, the intensity value is normalized to the range of [0, 1], and the normalized intensity value is NIn. Finally, the normalized intensity value is mapped to the gray value gv to generate a grayscale image of the point cloud.

[0053] Based on the generated point cloud grayscale image, the Canny algorithm is used to extract the disease edge feature points and geometric edge feature points. The edge features extracted from the point cloud grayscale image are edge points GP. edg The set; for each edge point GP edg The corresponding projected coordinates are (u0, v0). A KD-tree is used to search for the nearest point in the original point cloud, and edge points GP are extracted from the original point cloud based on the KD-tree search results. edg The corresponding point cloud P edg Its three-dimensional coordinates are (x pe ,y pe ,z pe ).

[0054] Optionally, in the image edge extraction module, a grayscale image is first generated for the target image, and then the Canny algorithm is used to extract its lesion edge feature points and geometric edge feature points. The edge features extracted in the point cloud grayscale image are edge points GI. edg The set of each edge feature point GI edg The coordinates in the image coordinate system are (u ie ,v ie );

[0055] Based on the extracted feature points GI edg The set of edge points is used to construct a KD-tree-based data structure.

[0056] Optionally, the edge pairing module assumes an initial transformation matrix T from the point cloud coordinate system to the camera coordinate system. in T in From the rotation matrix R in Translation vector t in Composition, R in Given a 3×3 matrix, t in Given a 3×1 vector; based on the initial transformation matrix T in The extracted feature points P from the point cloud edg Transform to camera coordinate system to obtain Pedgc The corresponding camera coordinates are (x c ,y c ,z c );

[0057] P edgc =R in P edg +t in (1)

[0058] Based on the obtained feature point P edgc and the corresponding camera coordinates (x) c ,y c ,z c Based on the camera intrinsic parameter matrix K and distortion coefficients, the feature points in the extracted point cloud are projected onto the image plane; based on the coordinates (x, y) in the camera coordinate system... c ,y c ,z c The normalized planar coordinates are obtained as follows:

[0059]

[0060] Feature point P after considering camera distortion edgc The normalized plane coordinates are (x dist ,y dist ):

[0061]

[0062] Where k1, k2, and k3 are the radial distortion coefficients of the camera; p1 and p2 are the tangential distortion coefficients of the camera.

[0063] Based on the camera's intrinsic parameter matrix K, the feature points P are... edgc Transform to the image plane to obtain feature point P edgi Its corresponding pixel coordinates on the image plane are (u pe ,v pe ),

[0064]

[0065] Wherein, the camera intrinsic parameter matrix K is a 3×3 matrix:

[0066]

[0067] Among them, c x c y f is the coordinate of the optical center on the pixel plane. x f y These are the focal lengths that represent the image scaling along the x-axis and y-axis, respectively;

[0068] Point cloud feature points P edgc Transform the feature point P from the camera coordinate system to the image plane coordinate system. edgi The process, i.e. the process from formula (2) to formula (7), is represented by the function π:

[0069] P edgi =π(P edgc (8)

[0070] According to formula (1), the feature points P of the point cloud edg Transform the edge feature point P from the point cloud coordinate system to the image plane coordinate system. edgi Represented as:

[0071] P edgi =π(R) in P edg +t in (9)

[0072] Based on the edge feature points P of the point cloud projected onto the camera plane edgi Coordinates (u) on the camera plane pe ,v pe ) and extracted image edge feature points GI edg Coordinates (u) on the camera plane ie ,v ie Perform nearest neighbor matching; for the j-th feature point P edgij Find its nearest edge point GI using the constructed KD tree. edgj And calculate the distance between them as d. j ;

[0073] d j =||P edgij -GI edgj || (10)

[0074] After calculating the distances to all projected points, take all distances d. j The sum of the first 90% of the distances is used to construct the optimization objective function:

[0075]

[0076] Where m is the feature point P edgi The number of midpoints;

[0077] According to formula (9), the optimization objective function can be obtained:

[0078]

[0079] Where R is the rotation matrix to be optimized; t is the translation vector to be optimized;

[0080] By optimizing the rotation matrix R and the translation vector t to minimize the value of the objective function, the final rotation matrix R is obtained. ul Translation vector t ul .

[0081] Based on the same inventive concept, the present invention provides an electronic device including a processor and a storage medium;

[0082] The storage medium is used to store instructions;

[0083] The processor is configured to operate according to the instructions to perform the steps of the method described above.

[0084] Based on the same inventive concept, the computer-readable storage medium of the present invention stores a computer program thereon, which, when executed by a processor, implements the steps of the method described above.

[0085] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: This invention combines the intensity information of point clouds to generate grayscale images and extracts the three-dimensional edge features of the disease point cloud using the Canny algorithm; compared to generating grayscale images using only coordinate information, it can extract more accurate edge information; compared to directly extracting edge features from point clouds and converting them into grayscale images, this method is more convenient and can more accurately extract the edge feature points of the disease for point cloud and image registration; This invention establishes an objective function by the distance between projection points and eliminates the influence of outliers on the registration results, further improving robustness; This invention minimizes the projection error by optimizing the objective function to obtain the rotation matrix and translation vector for the final point cloud-image registration, achieving high-precision registration based on edge features; This invention does not require a target and does not rely on manual annotation, automatically extracting three-dimensional edge features and achieving accurate matching with image edges, effectively solving the problem of large registration errors between heterogeneous data; At the same time, through multi-dimensional information fusion, namely the fusion of point cloud geometric features and image texture information, it provides reliable technical support for the spatial measurement and severity assessment of bridge diseases. Attached Figure Description

[0086] Figure 1 This is a flowchart of the present invention;

[0087] Figure 2 This is the grayscale image of the point cloud generated in this invention;

[0088] Figure 3 This is a schematic diagram of the edge features extracted from the grayscale image of the point cloud in this invention;

[0089] Figure 4 This is a schematic diagram of the corresponding points in the original point cloud for the point cloud edge features in this invention;

[0090] Figure 5This is a schematic diagram of the edge features extracted from the target image in this invention;

[0091] Figure 6 This is a schematic diagram illustrating the matching effect between point cloud features and image features in this invention;

[0092] Figure 7 This is a schematic diagram illustrating the final matching effect between point clouds and images in this invention. Detailed Implementation

[0093] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0094] like Figure 1 As shown, this invention discloses a point cloud-image registration method for spatial morphology detection of bridge defects, comprising the following steps:

[0095] S1. Point cloud processing and rotation alignment: Extract the normal vector of the principal plane of the point cloud using the RANSAC algorithm, and construct a rotation matrix to make the principal plane of the point cloud parallel to the XZ plane;

[0096] S2. Point cloud 2D projection and edge feature point extraction: Based on the intensity value of the point cloud, the point cloud rotated in step S1 is projected into a grayscale image in the main plane direction, and the Canny algorithm is used to extract the disease edge feature points and geometric edge feature points; the 3D edge feature points corresponding to the edge feature points are extracted from the original point cloud by searching the projection coordinates and KD-Tree.

[0097] Step S2 specifically includes the following steps:

[0098] S201. For the point cloud after rotating the main plane to be parallel to the XZ plane in step S1, extract the three-dimensional coordinates (x, y, z) and intensity value In from the point cloud data P, and project the three-dimensional point cloud onto the XZ plane. Construct a KD tree based on the projection point coordinates; then normalize the intensity value to the range [0, 1], and the normalized intensity value is NIn; finally, map the normalized intensity value to the grayscale value gv to generate a grayscale image of the point cloud, such as... Figure 2 As shown;

[0099] S202. Based on the point cloud grayscale image generated in step S201, the Canny algorithm is used to extract the disease edge feature points and geometric edge feature points. The edge features extracted in the point cloud grayscale image are edge points GP. edg A set, such as Figure 3 As shown; for each edge point GP edg The corresponding projected coordinates are (u0, v0). A KD-tree is used to search for the nearest point in the original point cloud, and edge points GP are extracted from the original point cloud based on the KD-tree search results. edg The corresponding point cloud Pedg ,like Figure 4 As shown, its three-dimensional coordinates are (x... pe ,y pe ,z pe );

[0100] S3. Image edge extraction and matching construction: First, a grayscale image is generated for the target image. Then, the Canny algorithm is used to extract the geometric edge features of the lesions and structures in the image and to construct an image edge KD tree to support fast matching.

[0101] Step S3 specifically includes the following steps:

[0102] S301. First, generate a grayscale image of the target image. Then, use the Canny algorithm to extract its disease edge feature points and geometric edge feature points. The edge features extracted from the point cloud grayscale image are edge points GI. edg A set, such as Figure 5 As shown, each edge feature point GI edg The coordinates in the image coordinate system are (u ie ,v ie );

[0103] S302, Based on the feature points GI extracted in step S301 edg The set of edge points is used to construct a KD-tree-based data structure, which provides a foundation for subsequent spatial matching and distance optimization between point cloud projection points and image edges;

[0104] S4. Initial projection and edge pairing of point cloud feature points: Combining initial extrinsic parameters, camera intrinsic parameters and distortion parameters, the 3D edge feature points of the point cloud extracted in step S2 are projected onto the target image plane and matched with the nearest neighbor of the image edge feature points extracted in step S3; with the average projection error as the target, a 6-DOF extrinsic parameter optimization function is constructed to optimize the rotation matrix and translation vector.

[0105] Step S4 specifically includes the following steps:

[0106] S401. Initial Projection of Point Cloud Feature Points: Assume the initial transformation matrix T from the point cloud coordinate system to the camera coordinate system. in T in From the rotation matrix R in Translation vector t in Composition, R in Given a 3×3 matrix, t in Given a 3×1 vector; based on the initial transformation matrix T in The extracted feature points P from the point cloud edg Transform to camera coordinate system to obtain P edgcThe corresponding camera coordinates are (x c ,y c ,z c );

[0107] P edgc =R in P edg +t in (1)

[0108] S402, Based on the feature point P obtained in step S401 edgc and the corresponding camera coordinates (x) c ,y c ,z c Based on the camera intrinsic parameter matrix K and distortion coefficients, the feature points in the extracted point cloud are projected onto the image plane; based on the coordinates (x, y) in the camera coordinate system... c ,y c ,z c The normalized planar coordinates are obtained as follows:

[0109]

[0110] Feature point P after considering camera distortion edgc The normalized plane coordinates are (x dist ,y dist ):

[0111]

[0112] Where k1, k2, and k3 are the radial distortion coefficients of the camera; p1 and p2 are the tangential distortion coefficients of the camera.

[0113] Based on the camera's intrinsic parameter matrix K, the feature points P are... edgc Transform to the image plane to obtain feature point P edgi Its corresponding pixel coordinates on the image plane are (u pe ,v pe ),

[0114]

[0115] Wherein, the camera intrinsic parameter matrix K is a 3×3 matrix:

[0116]

[0117] Among them, c x c y f is the coordinate of the optical center on the pixel plane. x f y These are the focal lengths that represent the image scaling along the x-axis and y-axis, respectively;

[0118] Point cloud feature points P edgc Transform the feature point P from the camera coordinate system to the image plane coordinate system. edgi The process, i.e. the process from formula (2) to formula (7), is represented by the function π:

[0119] P edgi =π(P edgc (8)

[0120] According to formula (1), the feature points P of the point cloud edg Transform the edge feature point P from the point cloud coordinate system to the image plane coordinate system. edgi It can be represented as:

[0121] P edgi =π(R) in P edg +t in (9)

[0122] S403, Based on the edge feature points P of the point cloud projected onto the camera plane. edgi Coordinates (u) on the camera plane pe ,v pe ) and the image edge feature points GI extracted in step S301 edg Coordinates (u) on the camera plane ie ,v ie Perform nearest neighbor matching; for the j-th feature point P edgij The nearest edge point GI is found using the KD tree established in step S302. edgj And calculate the distance between them as d. j ;

[0123] d j =||P edgij -GI edgj || (10)

[0124] After calculating the distances to all projected points, to increase robustness and reduce the impact of outliers on the registration results, all distances d are taken. j The sum of the first 90% of the distances is used to construct the optimization objective function:

[0125]

[0126] Where m is the feature point P edgi The number of midpoints;

[0127] According to formula (9), the optimization objective function can be obtained:

[0128]

[0129] Where R is the rotation matrix to be optimized; t is the translation vector to be optimized;

[0130] By optimizing the rotation matrix R and the translation vector t to minimize the value of the objective function, the final rotation matrix R is obtained. ul Translation vector t ul The matching effect of the optimized point cloud features and image features is as follows: Figure 6 As shown, red dots represent image edge feature points, and green dots represent pixel edge feature points.

[0131] S5. Using the optimized extrinsic parameters, the complete point cloud is projected onto the image plane to achieve final registration;

[0132] Using the optimized rotation matrix R ul Translation vector t ul The original point cloud P is projected onto the image plane to achieve final registration, such as... Figure 7 As shown.

[0133] Example 2: A point cloud-image registration system for spatial morphology detection of bridge defects according to the present invention includes:

[0134] The point cloud processing alignment module is used to extract the normal vector of the principal plane of the point cloud and construct a rotation matrix to make the principal plane of the point cloud parallel to the XZ plane.

[0135] The feature point extraction module is used to project the point cloud rotated in the point cloud processing and alignment module into a grayscale image in the main plane direction based on the intensity value of the point cloud, and extract the disease edge feature points and geometric edge feature points; and extract the three-dimensional edge feature points corresponding to the edge feature points in the original point cloud through projection coordinates and KD tree search.

[0136] In the feature point extraction module, the point cloud processing alignment module rotates the main plane to be parallel to the XZ plane, extracts the three-dimensional coordinates (x, y, z) and intensity value In from the point cloud data P, and projects the three-dimensional point cloud onto the XZ plane. A KD tree is constructed based on the coordinates of the projected points. Then, the intensity value is normalized to the range [0, 1], and the normalized intensity value is NIn. Finally, the normalized intensity value is mapped to the gray value gv to generate a grayscale image of the point cloud.

[0137] Based on the generated point cloud grayscale image, the Canny algorithm is used to extract the disease edge feature points and geometric edge feature points. The edge features extracted from the point cloud grayscale image are edge points GP. edg The set; for each edge point GP edg The corresponding projected coordinates are (u0, v0). A KD-tree is used to search for the nearest point in the original point cloud, and edge points GP are extracted from the original point cloud based on the KD-tree search results. edg The corresponding point cloud Pedg Its three-dimensional coordinates are (x pe ,y pe ,z pe ).

[0138] The image edge extraction module is used to first generate a grayscale image of the target image, and then extract the geometric edge features of the lesions and structures in it to construct an image edge KD tree;

[0139] In the image edge extraction module, a grayscale image is first generated for the target image. Then, the Canny algorithm is used to extract the lesion edge feature points and geometric edge feature points. The edge features extracted from the point cloud grayscale image are edge points GI. edg The set of each edge feature point GI edg The coordinates in the image coordinate system are (u ie ,v ie );

[0140] Based on the extracted feature points GI edg The set of edge points is used to construct a KD-tree-based data structure.

[0141] The edge pairing module combines initial extrinsic parameters, camera intrinsic parameters, and distortion parameters to project the 3D edge feature points of the point cloud extracted by the feature point extraction module onto the target image plane and perform nearest neighbor matching with the image edge feature points extracted by the image edge extraction module; with the average projection error as the objective, a 6-DOF extrinsic parameter optimization function is constructed to optimize the rotation matrix and translation vector;

[0142] In the edge pairing module, it is assumed that the initial transformation matrix T is from the point cloud coordinate system to the camera coordinate system. in T in From the rotation matrix R in Translation vector t in Composition, R in Given a 3×3 matrix, t in Given a 3×1 vector; based on the initial transformation matrix T in The extracted feature points P from the point cloud edg Transform to camera coordinate system to obtain P edgc The corresponding camera coordinates are (x c ,y c ,z c );

[0143] P edgc =R in P edg +t in (1)

[0144] Based on the obtained feature point P edgcand the corresponding camera coordinates (x) c ,y c ,z c Based on the camera intrinsic parameter matrix K and distortion coefficients, the feature points in the extracted point cloud are projected onto the image plane; based on the coordinates (x, y) in the camera coordinate system... c ,y c ,z c The normalized planar coordinates are obtained as follows:

[0145]

[0146] Feature point P after considering camera distortion edgc The normalized plane coordinates are (x dist ,y dist ):

[0147]

[0148] Where k1, k2, and k3 are the radial distortion coefficients of the camera; p1 and p2 are the tangential distortion coefficients of the camera.

[0149] Based on the camera's intrinsic parameter matrix K, the feature points P are... edgc Transform to the image plane to obtain feature point P edgi Its corresponding pixel coordinates on the image plane are (u pe ,v pe ),

[0150]

[0151] Wherein, the camera intrinsic parameter matrix K is a 3×3 matrix:

[0152]

[0153] Among them, c x c y f is the coordinate of the optical center on the pixel plane. x f y These are the focal lengths that represent the image scaling along the x-axis and y-axis, respectively;

[0154] Point cloud feature points P edgc Transform the feature point P from the camera coordinate system to the image plane coordinate system. edgi The process, i.e. the process from formula (2) to formula (7), is represented by the function π:

[0155] P edgi =π(P edgc (8)

[0156] According to formula (1), the feature points P of the point cloud edgTransform the edge feature point P from the point cloud coordinate system to the image plane coordinate system. edgi Represented as:

[0157] P edgi =π(R) in P edg +t in (9)

[0158] Based on the edge feature points P of the point cloud projected onto the camera plane edgi Coordinates (u) on the camera plane pe ,v pe ) and the image edge feature points GI extracted in step S301 edg Coordinates (u) on the camera plane ie ,v ie Perform nearest neighbor matching; for the j-th feature point P edgij The nearest edge point GI is found using the KD tree established in step S302. edgj And calculate the distance between them as d. j ;

[0159] d j =||P edgij -GI edgj || (10)

[0160] After calculating the distances to all projected points, take all distances d. j The sum of the first 90% of the distances is used to construct the optimization objective function:

[0161]

[0162] Where m is the feature point P edgi The number of midpoints;

[0163] According to formula (9), the optimization objective function can be obtained:

[0164]

[0165] Where R is the rotation matrix to be optimized; t is the translation vector to be optimized;

[0166] By optimizing the rotation matrix R and the translation vector t to minimize the value of the objective function, the final rotation matrix R is obtained. ul Translation vector t ul .

[0167] The projection registration module is used to project the complete point cloud onto the image plane using optimized extrinsic parameters, achieving final registration. This is achieved using the optimized rotation matrix R. ul Translation vector t ulThe original point cloud P is projected onto the image plane to achieve final registration.

[0168] Example 3: An electronic device according to the present invention includes a processor and a storage medium;

[0169] The storage medium is used to store instructions;

[0170] The processor is configured to operate according to the instructions to perform the steps of the method described above.

[0171] Example 4: The computer-readable storage medium of the present invention stores a computer program thereon, which, when executed by a processor, implements the steps of the method described above.

Claims

1. A point cloud-image registration method for spatial morphology detection of bridge defects, characterized in that, Includes the following steps: S1. Extract the normal vector of the principal plane of the point cloud and construct a rotation matrix to make the principal plane of the point cloud parallel to the XZ plane; S2. Based on the intensity values ​​of the point cloud, project the rotated point cloud from step S1 onto the main plane as a grayscale image, and extract the disease edge feature points and geometric edge feature points; extract the three-dimensional edge feature points corresponding to the edge feature points in the original point cloud through projection coordinates and KD tree search; step S2 specifically includes the following steps: S201. For the point cloud after rotating the main plane to be parallel to the XZ plane in step S1, extract the three-dimensional coordinates (x, y, z) and intensity value In from the point cloud data P, and project the three-dimensional point cloud onto the XZ plane. Construct a KD tree based on the coordinates of the projected points. Then, normalize the intensity value to the range of [0, 1], and the normalized intensity value is NIn. Finally, map the normalized intensity value to the gray value gv to generate a grayscale image of the point cloud. S202. Based on the point cloud grayscale image generated in step S201, the Canny algorithm is used to extract the disease edge feature points and geometric edge feature points. The edge features extracted in the point cloud grayscale image are edge points GP. edg The set; for each edge point GP edg The corresponding projected coordinates are (u0, v0). A KD-tree is used to search for the nearest point in the original point cloud, and edge points (GP) are extracted from the original point cloud based on the KD-tree search results. edg Corresponding point cloud feature points Its three-dimensional coordinates are (x pe ,y pe ,z pe ); S3. First, generate a grayscale image of the target image, then extract the geometric edge features of the lesions and structures to obtain the image edge feature points GI. edg Construct a KD tree for image edges; S4. Combining the initial extrinsic parameters, camera intrinsic parameters, and distortion parameters, the 3D edge feature points of the point cloud extracted in step S2 are projected onto the target image plane and matched with the nearest neighbor of the image edge feature points extracted in step S3. With the average projection error as the objective, a 6-DOF extrinsic parameter optimization function is constructed to optimize the rotation matrix and translation vector. Step S4 specifically includes the following steps: The point cloud feature points extracted in step S2 Edge feature points transformed into image plane coordinate system ; S403, Based on the edge feature points of the point cloud projected onto the camera plane. Coordinates on the camera plane Image edge feature points extracted in step S301 Coordinates on the camera plane Perform nearest neighbor matching; for the j-th feature point The nearest edge point is found using the KD tree established in step S302. And calculate the distance between the two as ; （10）， After calculating the distances to all projected points, take all distances. The objective function is constructed from the sum of the first 90% of the distances. （11）， in, For feature points The number of midpoints; According to formula (9), the optimization objective function can be obtained: （12）， in, Let be the rotation matrix to be optimized; Let be the translation vector to be optimized; By optimizing the rotation matrix Translation vector To minimize the value of the objective function, we obtain the final rotation matrix R. ul Translation vector t ul ; S5. Using the optimized extrinsic parameters, the complete point cloud is projected onto the image plane to achieve final registration.

2. The point cloud-image registration method for spatial morphology detection of bridge defects according to claim 1, characterized in that: Step S3 specifically includes the following steps: S301. First, generate a grayscale image of the target image. Then, use the Canny algorithm to extract its disease edge feature points and geometric edge feature points. The edge features extracted from the point cloud grayscale image are edge points GI. edg The set of each edge feature point GI edg The coordinates in the image coordinate system are (u ie ,v ie ); S302, Based on the feature points GI extracted in step S301 edg The set of edge points is used to construct a KD-tree-based data structure.

3. The point cloud-image registration method for spatial morphology detection of bridge defects according to claim 1, characterized in that: In step S4, the point cloud feature points extracted in step S2 are... Edge feature points transformed into image plane coordinate system Specifically, it includes the following steps: S401. Assume the initial transformation matrix T from the point cloud coordinate system to the camera coordinate system. in T in By rotation matrix Translation vector composition, It is a 3×3 matrix. Given a 3×1 vector; based on the initial transformation matrix T in Extract feature points from the point cloud Transform to camera coordinate system to obtain The corresponding coordinates in the camera coordinate system are ; （1）， S402, Based on the feature points obtained in step S401 and the corresponding coordinates in the camera coordinate system The feature points extracted from the point cloud are projected onto the image plane based on the camera intrinsic parameter matrix K and distortion coefficients; the coordinates in the camera coordinate system are then used to... The normalized plane coordinates are obtained as follows: （2）， Feature points after considering camera distortion effects The normalized plane coordinates are : （3）， （4）， （5）， in, The radial distortion coefficient of the camera; The tangential distortion coefficient of the camera; Based on the camera's intrinsic parameter matrix K, the feature points are... Transform to the image plane to obtain feature points Its corresponding pixel coordinates on the image plane are , （6）， Wherein, the camera intrinsic parameter matrix K is a 3×3 matrix: （7）， in, and These are the coordinates of the optical center on the pixel plane. and They respectively represent along shaft and The focal length of the image scaling along the axis; Point cloud feature points Feature points transformed from camera coordinate system to image plane coordinate system The process, i.e. the process from formula (2) to formula (7), is represented by the function π: （8）， According to formula (1), the feature points of the point cloud Transformation from point cloud coordinate system to image plane coordinate system for edge feature points Represented as: （9）。 4. A point cloud-image registration system for spatial morphological detection of bridge defects, characterized in that, include: The point cloud processing alignment module is used to extract the normal vector of the principal plane of the point cloud and construct a rotation matrix to make the principal plane of the point cloud parallel to the XZ plane. The feature point extraction module is used to project the point cloud rotated in the point cloud processing and alignment module into a grayscale image in the main plane direction based on the intensity value of the point cloud, and extract the disease edge feature points and geometric edge feature points; and extract the three-dimensional edge feature points corresponding to the edge feature points in the original point cloud through projection coordinates and KD tree search. In the feature point extraction module, after rotating the main plane to be parallel to the XZ plane, the three-dimensional coordinates (x, y, z) and intensity value In of the point cloud data P are extracted, and the three-dimensional point cloud is projected onto the XZ plane. A KD tree is constructed based on the coordinates of the projected points. Then, the intensity value is normalized to the range [0, 1], and the normalized intensity value is NIn. Finally, the normalized intensity value is mapped to the gray value gv to generate a grayscale image of the point cloud. Based on the generated point cloud grayscale image, the Canny algorithm is used to extract the disease edge feature points and geometric edge feature points. The edge features extracted from the point cloud grayscale image are edge points GP. edg The set; for each edge point GP edg The corresponding projected coordinates are (u0, v0). A KD-tree is used to search for the nearest point in the original point cloud, and edge points (GP) are extracted from the original point cloud based on the KD-tree search results. edg Corresponding point cloud feature points Its three-dimensional coordinates are (x pe ,y pe ,z pe ); The image edge extraction module first generates a grayscale image of the target image, and then extracts the geometric edge features of the lesions and structures to obtain image edge feature points (GI). edg Construct a KD tree for image edges; The edge pairing module combines initial extrinsic parameters, camera intrinsic parameters, and distortion parameters to project the 3D edge feature points of the point cloud extracted by the feature point extraction module onto the target image plane and perform nearest neighbor matching with the image edge feature points extracted by the image edge extraction module; with the average projection error as the objective, a 6-DOF extrinsic parameter optimization function is constructed to optimize the rotation matrix and translation vector; The edge matching module will extract point cloud feature points. Edge feature points transformed into image plane coordinate system Based on the edge feature points of the point cloud projected onto the camera plane Coordinates on the camera plane With extracted image edge feature points Coordinates on the camera plane Perform nearest neighbor matching; for the j-th feature point The nearest edge point is found using the KD tree established in step S302. And calculate the distance between the two as ; （10）， After calculating the distances to all projected points, take all distances. The objective function is constructed from the sum of the first 90% of the distances. （11）， in, For feature points The number of midpoints; According to formula (9), the optimization objective function can be obtained: （12）， in, Let be the rotation matrix to be optimized; Let be the translation vector to be optimized; By optimizing the rotation matrix Translation vector To minimize the value of the objective function, we obtain the final rotation matrix R. ul Translation vector t ul ; The projection registration module is used to project the complete point cloud onto the image plane using optimized extrinsic parameters to achieve final registration.

5. A point cloud-image registration system for spatial morphology detection of bridge defects according to claim 4, characterized in that: In the image edge extraction module, a grayscale image is first generated for the target image. Then, the Canny algorithm is used to extract the lesion edge feature points and geometric edge feature points. The edge features extracted from the point cloud grayscale image are edge points GI. edg The set of each edge feature point GI edg The coordinates in the image coordinate system are (u ie ,v ie ); Based on the extracted feature points GI edg The set of edge points is used to construct a KD-tree-based data structure.

6. A point cloud-image registration system for spatial morphology detection of bridge defects according to claim 5, characterized in that: The edge matching module will extract point cloud feature points. Edge feature points transformed into image plane coordinate system Specifically: Assume the initial transformation matrix T from the point cloud coordinate system to the camera coordinate system in T in From the rotation matrix R in Translation vector t in Composition, R in Given a 3×3 matrix, t in Given a 3×1 vector; based on the initial transformation matrix T in The extracted feature points P from the point cloud edg Transform to camera coordinate system to obtain P edgc The corresponding coordinates in the camera coordinate system are (x c ,y c ,z c ); （1）， Based on the obtained feature point P edgc and the corresponding camera coordinates (x) c ,y c ,z c The feature points extracted from the point cloud are projected onto the image plane based on the camera intrinsic parameter matrix K and distortion coefficients; the feature points are then projected onto the image plane based on the coordinates (x, y) in the camera coordinate system. c ,y c ,z c The normalized planar coordinates are obtained as follows: （2）， Feature point P after considering camera distortion edgc The normalized plane coordinates are (x dist ,y dist ): （3）， （4）， （5）， Where, 𝑘1, 𝑘2, 𝑘3 are the radial distortion coefficients of the camera; 𝑝1, 𝑝2 are the tangential distortion coefficients of the camera; Based on the camera's intrinsic parameter matrix K, the feature points P are... edgc Transform to the image plane to obtain feature point P edgi Its corresponding pixel coordinates on the image plane are (u pe ,v pe ), （6）， Wherein, the camera intrinsic parameter matrix K is a 3×3 matrix: （7）， Among them, c x c y f is the coordinate of the optical center on the pixel plane. x f y These are the focal lengths that represent the image scaling along the x-axis and y-axis, respectively; Point cloud feature points P edgc Transform the feature point P from the camera coordinate system to the image plane coordinate system. edgi The process, i.e. the process from formula (2) to formula (7), is represented by the function π: （8）， According to formula (1), the feature points P of the point cloud edg Transform the edge feature point P from the point cloud coordinate system to the image plane coordinate system. edgi Represented as: （9）。 7. An electronic device, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 3.

Images