Non-calibration local deformation detection method based on image-point cloud feature matching
By combining feature matching methods of 3D point clouds and 2D images, calibration-free local deformation detection was achieved, solving the problems of traditional methods relying on camera calibration and the limited generalization ability of learning-based methods, and realizing high-precision and automated local deformation measurement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-08
- Publication Date
- 2026-04-07
AI Technical Summary
In the monitoring of local deformation of concrete structures, the accuracy of existing technologies such as 3D laser scanning is limited by point cloud resolution and noise. Traditional image measurement methods rely on cumbersome and time-consuming camera calibration, while learning-based methods require a large amount of labeled data and have limited generalization ability, making it impossible to achieve accurate local deformation detection without calibration.
By acquiring high-precision 3D point clouds and 2D images, an initial depth map is generated using a monocular depth estimation model and scaled. Rotation-invariant feature descriptors are extracted for feature matching, camera parameters are estimated, and point cloud data is projected. Combined with an image segmentation model, the region of interest is automatically segmented to achieve calibration-free local deformation detection.
It achieves millimeter-level precision measurement without camera calibration, has a high degree of automation, is applicable to various concrete structures, can detect local deformations such as crack width and rebar displacement, reduces technical barriers and on-site operation time, and has strong robustness and wide applicability.
Smart Images
Figure CN121810786A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the interdisciplinary field of structural health monitoring and computer vision, and in particular to a calibration-free local deformation detection method based on image-point cloud feature matching. Background Technology
[0002] In the field of civil engineering, deformation monitoring of concrete structures (such as pipes, bridges, and buildings) is crucial. Structural deformation can be divided into overall deformation (such as deflection and tilt) and local deformation (such as crack propagation and rebar slippage). Currently, 3D laser scanning technology can efficiently acquire the overall point cloud model of a structure and accurately calculate the overall deformation by comparing point clouds from different periods. However, point cloud data has inherent limitations in characterizing fine local features at the millimeter or even sub-millimeter level; its accuracy is limited by point cloud resolution and noise levels.
[0003] On the other hand, high-resolution digital cameras can clearly capture these fine local features, but traditional image-based measurement methods heavily rely on rigorous camera calibration (obtaining precise intrinsic and extrinsic parameters) to establish a mapping between image pixels and real-world dimensions. In actual engineering settings, performing precise camera calibration for each shooting location is often impractical, cumbersome, and time-consuming, greatly limiting the widespread application of this technology.
[0004] Existing research attempts to fuse point cloud and image data, but most rely on learning-based methods, requiring large amounts of labeled data for training and exhibiting limited generalization ability. Furthermore, most methods still depend on explicit camera calibration. Therefore, there is an urgent need to develop a method that can directly and accurately quantify local deformations in images without prior calibration, utilizing point clouds as a spatial reference. This method has significant theoretical value and broad engineering application prospects. Summary of the Invention
[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a calibration-free local deformation detection method based on image-point cloud feature matching. This invention can fully utilize the overall geometric accuracy of point clouds and the local detail resolution of images to achieve millimeter-level accurate measurement of local deformations such as crack width and rebar displacement on structural surfaces.
[0006] The objective of this invention is achieved through the following technical solution: a calibration-free local deformation detection method based on image-point cloud feature matching, comprising the following steps: (1) Acquire high-precision three-dimensional point cloud data of the structure under test in the deformed state and two-dimensional images from the same viewpoint; (2) Input the two-dimensional image into the pre-trained monocular depth estimation model to generate the corresponding initial depth map, and convert it into an initial three-dimensional point cloud through back projection of the camera model; calculate the scale difference between the initial three-dimensional point cloud and the high-precision three-dimensional point cloud data, and make the scales of the two consistent by scaling to obtain the image-generated point cloud after scale unification. (3) Extract rotation-invariant feature descriptors from the scale-unified image-generated point cloud and high-precision 3D point cloud data respectively to perform feature matching, establish point cloud matching pairs, and use the random sampling consensus algorithm to estimate the rigid body transformation required to register the image-generated point cloud to the high-precision 3D point cloud data, which is the camera's extrinsic parameter matrix; at the same time, directly infer the camera's intrinsic parameter matrix from the monocular depth estimation model. (4) Project the high-precision 3D point cloud data onto the virtual camera plane using the camera's extrinsic and intrinsic parameters to generate a depth map aligned with the 2D image viewpoint; use the image segmentation model SAM2 to extract common feature regions and feature point pairs from the 2D image and the generated depth map respectively, in order to calculate the 2D-2D transformation matrix from the 2D image coordinate system to the depth map coordinate system. (5) The user manually selects or automatically segments the region of interest representing local deformation on the two-dimensional image using the image segmentation model SAM2; using the 2D-2D transformation matrix, the pixel coordinates of the region of interest are mapped onto the depth map, and then the depth values of each pixel in the depth map and the intrinsic and extrinsic parameters of the camera are combined to calculate the actual coordinates of the key points in the region of interest in three-dimensional space through three-dimensional back projection; finally, the actual physical size of the local deformation is calculated based on these actual coordinates.
[0007] Furthermore, step (1) specifically includes: A 3D laser scanner is used to acquire high-precision 3D point cloud data of the structure under test after deformation. A digital camera is used to acquire a 2D image from the same viewpoint as the point cloud data, ensuring that the point cloud data and the image cover the same structural area.
[0008] Furthermore, the monocular depth estimation model adopts the Depth Pro model.
[0009] Furthermore, the calculation of the scale difference between the initial 3D point cloud and the high-precision 3D point cloud data, and the scaling to make the two scales consistent, specifically includes: The scale difference between the initial 3D point cloud and the high-precision 3D point cloud data is calculated to obtain the scale uniformity factor, which is expressed as follows: In the formula, represents the scale uniformity factor; d represents the spatial dimension index, which is taken sequentially along the three coordinate axes x, y, z. This represents the span of high-precision 3D point cloud data in the d-dimensional dimension. This represents the span of the initial 3D point cloud in dimension d; Multiply the initial 3D point cloud by a scale uniformity factor to complete scale alignment and obtain the scale-uniformed image generated point cloud.
[0010] Furthermore, step (3) specifically includes the following sub-steps: (3.1) A descriptor framework based on group isovariant feature learning is adopted to extract rotation-invariant feature descriptors from the scale-unified image to generate point clouds and high-precision 3D point cloud data respectively, and feature matching is performed to obtain a set of initial corresponding key point cloud pairs, i.e. point cloud matching pairs. (3.2) Based on point cloud matching pairs, it is assumed that the matched key points are in the image generated point cloud. The set in is In high-precision 3D point cloud data The set in is ;right and Centralized processing is performed to obtain the corresponding centralized key point set. and and based on and Construct the covariance matrix; (3.3) Perform singular value decomposition on the covariance matrix to determine the rigid body transformation required to register the image-generated point cloud to high-precision 3D point cloud data, including rotation matrix and translation vector; (3.4) A subset of point cloud matching pairs is randomly selected using the random sampling consensus algorithm and solved iteratively to obtain the optimal rotation matrix and the optimal translation vector; (3.5) Construct the camera's extrinsic matrix based on the optimal rotation matrix and the optimal translation vector; at the same time, directly infer the camera's intrinsic matrix from the monocular depth estimation model.
[0011] Furthermore, step (3.4) specifically includes: Suppose that the rotation matrix and translation vector calculated in the k-th iteration are respectively and ; First, the score based on the alignment error in 3D space is calculated by counting the number of times the distance between the source point and the target point is less than a threshold after statistical transformation: In the formula, This indicates a score based on alignment error in three-dimensional space; , Let represent the candidate rotation matrix and translation vector in the k-th iteration, respectively; Represents the 3D coordinates after the source point transformation; Represents the set of 3D coordinates of the initial point; Represents the set of 3D coordinates of the target point; Represents the distance threshold in 3D space; Then, a score based on the two-dimensional image plane alignment error is calculated by counting the number of projection points whose distance from the original image points is less than a threshold. In the formula, This represents a score based on the planar alignment error of a two-dimensional image. This represents the 2D coordinates of the transformed source keypoints projected onto the image plane; Represents the transformed source keypoints Z-axis coordinates in the camera coordinate system; Represents the two-dimensional pixel coordinates of the i-th keypoint in the original image; Represents the distance threshold in 2D space; Subsequently, through the study and The final comprehensive score is calculated by weighted summation: In the formula, This indicates the final overall score. For use in balancing and Weight parameters; Finally, the rotation matrix and translation vector with the highest overall score are selected as the optimal rotation matrix obtained through iterative solution. and optimal translation vector .
[0012] Furthermore, step (4) specifically includes: Using the camera's extrinsic and intrinsic parameters, high-precision 3D point cloud data is projected onto the virtual camera plane through the principle of 3D perspective projection, generating a depth map aligned with the 2D image viewpoint. Using the image segmentation model SAM2, common feature regions and feature point pairs are extracted from the 2D image and the generated depth map, respectively. Based on the feature regions and feature point pairs, a 2D-2D transformation matrix from the 2D image coordinate system to the depth map coordinate system is constructed. The transformation parameters are solved by the least squares method to obtain the solved 2D-2D transformation matrix.
[0013] Furthermore, the projection of high-precision 3D point cloud data onto the virtual camera plane using the principle of 3D perspective projection specifically includes: Assuming high-precision 3D point cloud data a certain point in the middle First, transform point P to the virtual camera coordinate system: In the formula, This represents the homogeneous coordinates of point P in the virtual camera coordinate system. This represents the three-dimensional coordinates of point P in the virtual camera coordinate system; then... Projected onto the image plane: In the formula, This represents the three-dimensional coordinates of point P before dehomogenization after it is projected onto the image plane. The final generated depth map has the following coordinates and depth values: In the formula, This represents the pixel coordinates of point P in the generated depth map. This represents the depth value of the corresponding pixel in the generated depth map for point P.
[0014] Furthermore, in step (5), the local deformation includes the width of the concrete crack, the amount of rebar retraction, and the size of the protective layer peeling off.
[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) No calibration feature: This invention completely eliminates the tedious traditional camera calibration steps, eliminating the need for pre-calibration of the camera, greatly reducing the technical threshold and on-site operation time, and improving practicality and efficiency.
[0016] (2) High-precision measurement: This invention uses high-precision laser point cloud as spatial size reference, which overcomes the error caused by perspective and distortion in pure image measurement and can realize millimeter-level local deformation quantification.
[0017] (3) Strong robustness: The present invention adopts advanced depth estimation, rotation invariant feature descriptor and robust estimation algorithm, which can effectively cope with challenges such as illumination changes, partial occlusion and differences in the initial pose of point cloud and image.
[0018] (4) Automation and versatility: This invention combines deep learning models for feature extraction and segmentation, with a high degree of automation. It does not depend on specific structural types and is applicable to the detection of local damage in various concrete structures such as pipes, beams, slabs, and walls.
[0019] (5) Cross-modal fusion: This invention combines three-dimensional point cloud and two-dimensional image for non-destructive detection of local fine deformation of structure. It innovatively combines the absolute scale information of point cloud with the rich detail information of image, and establishes a connection through rigorous mathematical transformation, thus achieving complementary advantages. Attached Figure Description
[0020] Figure 1 This is a flowchart of the calibration-free local deformation detection method based on image-point cloud feature matching of the present invention; Figure 2 This is a schematic diagram illustrating the principle of generating a depth map by projecting point clouds using estimated camera parameters in this invention. Figure 3 This is a schematic diagram illustrating the present invention of feature matching between two-dimensional images and depth maps and establishing 2D-2D transformation relationships; Figure 4 This is a schematic diagram showing the results of mapping the local deformation areas (cracks and reinforcing bars) from an image to a point cloud and measuring their dimensions; wherein, Figure 4 (a) in the diagram is a schematic diagram of the overall deformation of the pipeline; Figure 4 (b) in the diagram is a distribution map of the deformation. Detailed Implementation
[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit this application.
[0022] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.
[0023] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to determination," or "includes." Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process or method. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0024] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.
[0025] See Figure 1 This invention provides a calibration-free local deformation detection method based on image-point cloud feature matching, specifically including the following steps: (1) Data acquisition and input: Acquire high-precision three-dimensional point cloud data of the structure under test in the deformed state and two-dimensional images from the same perspective.
[0026] Specifically, a 3D laser scanner is used to acquire high-precision 3D point cloud data of the structure under test (such as a prestressed concrete cylinder pipe, PCCP) after deformation. Simultaneously, a high-resolution digital camera is used to capture high-resolution digital 2D images from locations where local deformations (such as cracks or rebar retraction) can be clearly observed. It is important to ensure that the acquired point cloud data and images cover the same structural area during data acquisition.
[0027] The 3D laser scanner can be the FARO Focus S350 model, and the high-resolution digital camera can be the DAHENG MER2-2000-6GM model. Of course, other models of 3D laser scanners and high-resolution digital cameras can also be used according to actual needs.
[0028] (2) Point cloud generation and scale unification based on deep learning: The two-dimensional image obtained in step (1) is input into the pre-trained monocular depth estimation model to generate the corresponding initial depth map, and it is converted into an initial three-dimensional point cloud by back projection of the camera model; the scale difference between the initial three-dimensional point cloud and the high-precision three-dimensional point cloud data obtained in step (1) is calculated, and the scale of the two is made consistent by scaling, so as to obtain the scale-unified image-generated point cloud.
[0029] Furthermore, the pre-trained monocular depth estimation model can employ the existing publicly available pre-trained DepthPro model. This model can predict depth information from the input 2D image, obtain the corresponding initial depth map, and then process the initial depth map through backprojection from the camera model to generate a preliminary 3D point cloud, which is the initial 3D point cloud. .
[0030] Furthermore, the scale difference between the initial 3D point cloud and the high-precision 3D point cloud data is calculated, and the scales are made consistent by scaling. Specifically, this includes: first calculating the initial 3D point cloud... The high-precision 3D point cloud data obtained in step (1) The scale difference between the two is used to obtain a scale uniformity factor. Specifically, the span of the initial 3D point cloud and the input 3D point cloud data in the x, y, and z coordinate axes is calculated respectively, and the scaling ratio in each axis direction is obtained. The final scale uniformity factor is the arithmetic mean of the scaling ratios in the x, y, and z directions. The formula for calculating the scale uniformity factor can be expressed as: In the formula, represents the scale uniformity factor; d represents the spatial dimension index, which is taken sequentially along the three coordinate axes x, y, z. This refers to the high-precision 3D point cloud data obtained in step (1). The span in the d-dimensional dimension, i.e., high-precision 3D point cloud data The difference between the maximum and minimum values in dimension d; This represents the initial 3D point cloud generated from a 2D image. The span in dimension d, i.e., the initial 3D point cloud The difference between the maximum and minimum values in the d-dimensional dimension. Then, the initial 3D point cloud generated from the 2D image is multiplied by a scale uniformity factor. After completing scale alignment, a point cloud of the image with uniform scale is obtained. This ensures that the scale of the point cloud generated from the image is consistent with the scale of the real world.
[0031] (3) Cross-modal feature matching and camera parameter estimation: Rotation-invariant feature descriptors are extracted from the scale-unified image-generated point cloud and high-precision 3D point cloud data respectively for feature matching, point cloud matching pairs are established, and the Random Sample Consensus (RANSAC) algorithm is used to robustly estimate the rigid body transformation required to register the image-generated point cloud to the high-precision 3D point cloud data. This rigid body transformation defines the extrinsic parameter matrix E of the camera that captures the 2D image, so this rigid body transformation is the extrinsic parameter matrix E of the camera; at the same time, the intrinsic parameter matrix K of the camera is directly inferred from the monocular depth estimation model.
[0032] It should be noted that, in order to estimate the camera pose relative to the high-precision 3D point cloud data when the image was captured, it is necessary to establish a scale-uniform image generation point cloud. The three-dimensional point cloud data obtained in step (1) The correspondence between them is used to obtain point cloud matching pairs.
[0033] Furthermore, cross-modal feature matching and camera parameter estimation specifically include the following sub-steps: (3.1) A descriptor framework based on group isovariant feature learning (such as YOHO) is used to generate point clouds from the scale-unified images. and high-precision 3D point cloud data Rotation-invariant feature descriptors are extracted to ensure robust matching under different viewpoints and point cloud densities. Feature matching is then performed to obtain an initial set of corresponding key point cloud pairs, i.e., point cloud matching pairs. The key point clouds should be selected from undeformed regions of the structure to avoid introducing incorrect correspondences from deformed regions and ensure the accuracy of subsequent parameter estimation.
[0034] (3.2) Based on point cloud matching pairs, it is assumed that the matched key points are in the image generated point cloud. The set in is In high-precision 3D point cloud data The set in is ; then the set of two point clouds and Centralized processing is performed to obtain the corresponding centralized key point set. and And based on two centralized key point sets and Construct the covariance matrix C: In the formula, C represents the covariance matrix, and n represents the number of matching pairs in the corresponding point cloud. express The coordinate vector (3×1) of the i-th centered high-precision 3D point cloud data. express The coordinate vector (3×1) of the point cloud generated from the i-th centered image is given, where the superscript T indicates the transpose of the vector.
[0035] (3.3) By performing singular value decomposition (SVD) on the covariance matrix C, the rigid body transformation required to register the image-generated point cloud to high-precision 3D point cloud data can be determined, including the rotation matrix R and the translation vector t: In the formula, U represents the matrix obtained after singular value decomposition of the covariance matrix C, where U and V are unitary matrices and Σ is a singular value diagonal matrix; R represents the estimated rotation matrix (3×3), which describes the rotation relationship from the image-generated point cloud coordinate system to the high-precision 3D point cloud coordinate system; t represents the estimated translation vector (3×1), which describes the translation relationship from the image-generated point cloud coordinate system to the high-precision 3D point cloud coordinate system. This represents the centroid coordinates of all matching key points in high-precision 3D point cloud data. This represents the centroid coordinates of all matching keypoints in the generated point cloud of the image.
[0036] (3.4) During the feature matching process, there may be some erroneous feature correspondences. To solve this problem, the Random Sample Consensus Algorithm (RANSAC) is used to randomly select a subset of point cloud matching pairs and iteratively solve the problem to determine the optimal rigid body transformation matrix, including the optimal rotation matrix and the optimal translation vector.
[0037] Specifically, the solution for the optimal rotation matrix and optimal translation vector is achieved through the following method: Let the rigid body transformation calculated in a certain iteration be... and Its optimization process follows two principles: The first criterion is 3D spatial alignment accuracy, which is calculated by statistically analyzing the number of times the distance between the source point and the target point after transformation is less than a threshold to determine the score based on the 3D spatial alignment error. In the formula, This indicates a score based on alignment error in three-dimensional space; , Let represent the candidate rotation matrix and translation vector in the k-th RANSAC iteration, respectively; Represents the 3D coordinates after the source point transformation; Represents the set of 3D coordinates of the initial point; Represents the set of 3D coordinates of the target point; This represents the distance threshold in 3D space.
[0038] The second criterion is: 2D image planar alignment accuracy, which is calculated by counting the number of times the distance between the projected point and the original image point is less than a threshold to determine the score based on the 2D image planar alignment error. In the formula, This represents a score based on the planar alignment error of a two-dimensional image. This represents the 2D coordinates of the transformed source keypoints projected onto the image plane; Represents the transformed source keypoints Z-axis coordinates (depth value) in the camera coordinate system; Represents the two-dimensional pixel coordinates of the i-th keypoint in the original image; This represents the distance threshold in 2D space.
[0039] Subsequently, through the study and The final comprehensive score is calculated by weighted summation: In the formula, This indicates the final overall score. For use in balancing and The weight parameter has a value range of 0 to 1.
[0040] Finally, the rotation matrix and translation vector with the highest overall score are selected as the optimal rotation matrix obtained through iterative solution. and optimal translation vector .
[0041] It should be noted that this is achieved by optimizing the comprehensive score, which is a weighted sum of the alignment error in 3D space and the alignment error of the 2D image plane, to ensure that the estimated transformation has high accuracy in both 3D space and 2D projection.
[0042] (3.5) Construct the camera's extrinsic parameter matrix E based on the optimal rotation matrix and optimal translation vector, denoted as: Simultaneously, the camera's intrinsic parameter matrix K is directly inferred from the monocular depth estimation model.
[0043] (4) Point cloud projection and image-depth map feature alignment: Using the camera's extrinsic matrix E and intrinsic matrix K obtained in step (3), the high-precision 3D point cloud data is projected onto the virtual camera plane to generate a high-precision depth map aligned with the viewpoint of the input 2D image. Subsequently, the powerful image segmentation model SAM2 is used to extract common feature regions and feature point pairs on the input 2D image and the generated depth map, respectively. Based on the feature regions and feature point pairs, the 2D-2D transformation matrix from the 2D image coordinate system to the depth map coordinate system is calculated.
[0044] (4.1) Using the extrinsic parameter matrix E and intrinsic parameter matrix K of the camera obtained in step (3), the high-precision three-dimensional point cloud data is processed. By projecting a 3D perspective projection onto the imaging plane of a virtual camera, a high-precision depth map is generated that is strictly aligned with the viewpoint of the input 2D image. ,like Figure 2 As shown.
[0045] Specifically, assuming high-precision 3D point cloud data a certain point in the middle The image is projected onto the virtual camera's imaging plane using the principle of 3D perspective projection. The projection process is as follows: First, transform point P to the virtual camera coordinate system: In the formula, This represents the homogeneous coordinates of point P in the virtual camera coordinate system. This represents the three-dimensional coordinates of point P in the virtual camera coordinate system. Then... Projected onto the image plane: In the formula, This represents the three-dimensional coordinates of point P before dehomogenization after it is projected onto the image plane.
[0046] The final generated depth map, aligned with the viewpoint of the input 2D image, has the following coordinates and depth values: In the formula, This represents the pixel coordinates of point P in the generated depth map. This represents the depth value of the corresponding pixel in the generated depth map for point P, i.e., its position in the virtual camera coordinate system. coordinate.
[0047] (4.2) After generating the depth map, the general image segmentation model SAM2 is used to automatically identify and match common salient feature regions in the 2D image and depth map, thereby establishing a dense 2D-2D correspondence, such as... Figure 3As shown, the transformation matrix parameters are solved using the least squares method.
[0048] Specifically, generating depth maps Then, using the general image segmentation model SAM2, the original 2D image and depth map were segmented separately. Extract common salient feature regions and feature point pairs. Let the set of feature point pairs in the input 2D image be... Depth map The corresponding feature point pair set is Based on these correspondences, a 2D-2D transformation matrix can be constructed from the input 2D image to the generated depth map: In the formula, This represents the 2D-2D transformation matrix from a 2D image to a depth map. The transformation parameters are solved using the least squares method. .
[0049] (5) Local deformation quantization: The user manually selects or automatically segments the region of interest (ROI) representing local deformation (such as a crack, the end of a rebar, etc.) on the two-dimensional image using the image segmentation model SAM2; using the 2D-2D transformation matrix obtained in step (4), the pixel coordinates of the ROI are mapped onto the depth map, and then the depth values of each pixel in the depth map and the intrinsic and extrinsic parameters of the camera are combined to calculate the actual coordinates of the key points in the ROI in three-dimensional space through three-dimensional back projection; finally, the actual physical dimensions of the local deformation (such as crack width and rebar retraction length) are calculated based on these actual coordinates.
[0050] Furthermore, local deformations include, but are not limited to, concrete crack width, rebar retraction, and protective layer spalling. On the original two-dimensional image, corresponding region selection strategies are adopted for different types of local deformation: for cracks, a pre-trained crack image segmentation model is used for automatic identification and extraction to select the region of interest; for rebar retraction, manual region selection is used for measurement to select the region of interest.
[0051] Furthermore, using the transformation matrix obtained in step (4), the pixel coordinates of the selected region of interest are mapped to the depth map. Above. Each pixel coordinate (u, v) in the depth map is associated with a depth value d. The image coordinates and depth values are combined to form a point in the depth coordinate system. The three-dimensional coordinates of the object in the real-world coordinate system are calculated by three-dimensional back projection. The reverse projection process is the inverse operation of the projection process in step (4): First, calculate the coordinates in the camera coordinate system: Then transform to the world coordinate system: Perform the inverse projection described above on all points within the region of interest to obtain a set of 3D coordinates. Calculate the corresponding physical scale based on the type of local deformation: for rebar retraction, calculate the Euclidean distance between the two endpoints representing the initial and current positions of the rebar; for crack width, calculate the Euclidean distance between the corresponding point sets on either side of the crack.
[0052] For example, in a PCCP pipeline prototype experiment, the method described in this invention was applied to measure surface cracks and rebar shrinkage caused by wire breakage, and local deformation was observed as follows: Figure 4 As shown in (a) above, the distribution of deformation is as follows: Figure 4 As shown in (b) of the paper. Experimental results show that on a pipe with a diameter of about 2 meters, the maximum relative error in measuring the rebar shrinkage is 10.3% (average relative error 8.9%), and the maximum relative error in measuring the crack width is 13.1% (average relative error 10.7%), achieving millimeter-level measurement accuracy and verifying the effectiveness and practicality of the invention.
[0053] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A calibration-free local deformation detection method based on image-point cloud feature matching, characterized in that, Includes the following steps: (1) Acquire high-precision three-dimensional point cloud data of the structure under test in the deformed state and two-dimensional images from the same viewpoint; (2) Input the two-dimensional image into the pre-trained monocular depth estimation model to generate the corresponding initial depth map, and convert it into an initial three-dimensional point cloud through back projection of the camera model; Calculate the scale difference between the initial 3D point cloud and the high-precision 3D point cloud data, and make the scales of the two consistent by scaling to obtain a scale-unified image to generate a point cloud. (3) Extract rotation-invariant feature descriptors from the scale-unified image-generated point cloud and high-precision 3D point cloud data respectively to perform feature matching, establish point cloud matching pairs, and use the random sampling consensus algorithm to estimate the rigid body transformation required to register the image-generated point cloud to the high-precision 3D point cloud data, which is the camera's extrinsic parameter matrix; at the same time, directly infer the camera's intrinsic parameter matrix from the monocular depth estimation model. (4) Project the high-precision 3D point cloud data onto the virtual camera plane using the camera's extrinsic and intrinsic parameter matrices to generate a depth map aligned with the 2D image viewpoint; Using the image segmentation model SAM2, common feature regions and feature point pairs are extracted from the two-dimensional image and the generated depth map, respectively, to calculate the 2D-2D transformation matrix from the two-dimensional image coordinate system to the depth map coordinate system. (5) Users can manually select or automatically segment regions of interest representing local deformations on a two-dimensional image using the image segmentation model SAM2; using a 2D-2D transformation matrix, the pixel coordinates of the region of interest are mapped onto a depth map; and then, by combining the depth values of each pixel in the depth map with the camera's intrinsic and extrinsic parameters, the actual coordinates of key points in the region of interest in three-dimensional space are calculated through three-dimensional back projection. Ultimately, the actual physical dimensions of the local deformation are calculated based on these actual coordinates.
2. The calibration-free local deformation detection method based on image-point cloud feature matching according to claim 1, characterized in that, Step (1) specifically includes: A 3D laser scanner is used to acquire high-precision 3D point cloud data of the structure under test after deformation. A digital camera is used to acquire a 2D image from the same viewpoint as the point cloud data, ensuring that the point cloud data and the image cover the same structural area.
3. The calibration-free local deformation detection method based on image-point cloud feature matching according to claim 1, characterized in that, The monocular depth estimation model uses the Depth Pro model.
4. The calibration-free local deformation detection method based on image-point cloud feature matching according to claim 1, characterized in that, The calculation of the scale difference between the initial 3D point cloud and the high-precision 3D point cloud data, and the scaling to make the two scales consistent, specifically includes: The scale difference between the initial 3D point cloud and the high-precision 3D point cloud data is calculated to obtain the scale uniformity factor, which is expressed as follows: In the formula, represents the scale uniformity factor; d represents the spatial dimension index, which is taken sequentially along the three coordinate axes x, y, z. This represents the span of high-precision 3D point cloud data in the d-dimensional dimension. This represents the span of the initial 3D point cloud in dimension d; Multiply the initial 3D point cloud by a scale uniformity factor to complete scale alignment and obtain the scale-uniformed image generated point cloud.
5. The calibration-free local deformation detection method based on image-point cloud feature matching according to claim 1, characterized in that, Step (3) specifically includes the following sub-steps: (3.1) A descriptor framework based on group isovariant feature learning is adopted to extract rotation-invariant feature descriptors from the scale-unified image to generate point clouds and high-precision 3D point cloud data respectively, and feature matching is performed to obtain a set of initial corresponding key point cloud pairs, i.e. point cloud matching pairs. (3.2) Based on point cloud matching pairs, it is assumed that the matched key points are in the image generated point cloud. The set in is In high-precision 3D point cloud data The set in is ;right and Centralized processing is performed to obtain the corresponding centralized key point set. and and based on and Construct the covariance matrix; (3.3) Perform singular value decomposition on the covariance matrix to determine the rigid body transformation required to register the image-generated point cloud to high-precision 3D point cloud data, including rotation matrix and translation vector; (3.4) A subset of point cloud matching pairs is randomly selected using the random sampling consensus algorithm and solved iteratively to obtain the optimal rotation matrix and the optimal translation vector; (3.5) Construct the camera's extrinsic matrix based on the optimal rotation matrix and the optimal translation vector; at the same time, directly infer the camera's intrinsic matrix from the monocular depth estimation model.
6. The calibration-free local deformation detection method based on image-point cloud feature matching according to claim 5, characterized in that, Step (3.4) specifically includes: Suppose that the rotation matrix and translation vector calculated in the k-th iteration are respectively and ; First, the score based on the alignment error in 3D space is calculated by counting the number of times the distance between the source point and the target point is less than a threshold after statistical transformation: In the formula, This indicates a score based on alignment error in three-dimensional space; , Let represent the candidate rotation matrix and translation vector in the k-th iteration, respectively; Represents the 3D coordinates after the source point transformation; Represents the set of 3D coordinates of the initial point; Represents the set of 3D coordinates of the target point; Represents the distance threshold in 3D space; Then, a score based on the two-dimensional image plane alignment error is calculated by counting the number of projection points whose distance from the original image points is less than a threshold. In the formula, This represents a score based on the planar alignment error of a two-dimensional image. This represents the 2D coordinates of the transformed source keypoints projected onto the image plane; Represents the transformed source keypoints Z-axis coordinates in the camera coordinate system; Represents the two-dimensional pixel coordinates of the i-th keypoint in the original image; Represents the distance threshold in 2D space; Subsequently, through the study and The final comprehensive score is calculated by weighted summation: In the formula, This indicates the final overall score. For use in balancing and Weight parameters; Finally, the rotation matrix and translation vector with the highest overall score are selected as the optimal rotation matrix obtained through iterative solution. and optimal translation vector .
7. The calibration-free local deformation detection method based on image-point cloud feature matching according to claim 1, characterized in that, Step (4) specifically includes: Using the camera's extrinsic and intrinsic parameters, high-precision 3D point cloud data is projected onto the virtual camera plane through the principle of 3D perspective projection, generating a depth map aligned with the 2D image viewpoint. Using the image segmentation model SAM2, common feature regions and feature point pairs are extracted from the 2D image and the generated depth map, respectively. Based on the feature regions and feature point pairs, a 2D-2D transformation matrix from the 2D image coordinate system to the depth map coordinate system is constructed. The transformation parameters are solved by the least squares method to obtain the solved 2D-2D transformation matrix.
8. The calibration-free local deformation detection method based on image-point cloud feature matching according to claim 7, characterized in that, The process of projecting high-precision 3D point cloud data onto a virtual camera plane using the principle of 3D perspective projection specifically includes: Assuming high-precision 3D point cloud data a certain point in the middle First, transform point P to the virtual camera coordinate system: In the formula, This represents the homogeneous coordinates of point P in the virtual camera coordinate system. This represents the three-dimensional coordinates of point P in the virtual camera coordinate system; then... Projected onto the image plane: In the formula, This represents the three-dimensional coordinates of point P before dehomogenization after it is projected onto the image plane. The final generated depth map has the following coordinates and depth values: In the formula, This represents the pixel coordinates of point P in the generated depth map. This represents the depth value of the corresponding pixel in the generated depth map for point P.
9. The calibration-free local deformation detection method based on image-point cloud feature matching according to claim 1, characterized in that, In step (5), the local deformation includes the width of the concrete crack, the amount of rebar retraction, and the size of the protective layer peeling off.