A method and system for recognizing defects on the inner wall of a pipeline based on machine vision

Abstract

The present application belongs to the technical field of pipeline defect identification, and particularly relates to a pipeline inner wall defect identification method and system based on machine vision. The method comprises collecting an original image sequence of the pipeline inner wall, and acquiring odometer and inertial measurement unit data of a pipeline detection robot. The original image sequence is corrected for distortion according to the internal geometric parameters of the pipeline to obtain a corrected image sequence. The corrected image sequence is registered according to the odometer and inertial measurement unit data to obtain a panoramic development map of the pipeline inner wall. Defect identification is performed according to the panoramic development map. Self-calibration is performed through the circumferential characteristics of the pipeline inner wall, and odometer, inertial measurement unit and visual information are fused for joint optimization to improve the accuracy of pipeline inner wall defect imaging.

Description

Technical Field

[0001] This invention belongs to the field of pipeline defect recognition technology, specifically relating to a method and system for identifying pipeline inner wall defects based on machine vision. Background Technology

[0002] The vision system on a pipeline inspection robot is used to acquire images of the pipeline's inner wall, which is fundamental for identifying and locating pipeline defects. To expand the field of view coverage of a single image, inspection equipment often uses wide-angle lenses or fisheye lenses. The inner wall of the pipeline has a cylindrical curved surface, and the camera is positioned near the pipeline's central axis, shooting towards the pipe wall. The acquired image simultaneously includes lens optical distortion and surface projection distortion.

[0003] In the image distortion correction stage, mainstream calibration methods (such as Zhang Zhengyou's method) rely on placing calibration plates at multiple angles in the scene. This is almost impossible to operate in the extremely narrow, enclosed space of a pipe, making it difficult to accurately obtain calibration parameters. Even when using general fisheye distortion models for correction, these models do not consider the geometric characteristics of cylindrical curved surfaces, resulting in residual geometric distortion in the edge regions of the corrected image, such as distorting straight cracks into arcs. Correction distortion further exacerbates the difficulty of subsequent panoramic stitching. Stitching technology highly depends on stable and reliable feature point matching between images, but the surface texture of the inner wall of a pipe (such as PVC pipes or concrete pipes) is simple and sparse, resulting in a low feature matching success rate, ultimately leading to the inability to generate a panoramic image or the presence of severe misalignment.

[0004] Therefore, there is an urgent need for a machine vision-based method and system for identifying defects in the inner walls of pipes. Optimizing image distortion correction based on the internal spatial scene of the pipe can improve the success rate of defect identification. Summary of the Invention

[0005] (1) Technical problems to be solved The purpose of this invention is to provide a method and system for identifying defects in the inner wall of pipes based on machine vision, so as to solve the problem of low accuracy in identifying defects in the inner wall of pipes due to inaccurate image distortion correction and difficulty in feature registration in the pipeline environment.

[0006] (2) Technical solution To achieve the above objectives, in one aspect, the present invention provides a method for identifying defects in the inner wall of pipes based on machine vision, the method comprising: The system acquires raw image sequences of the inner wall of the pipe and obtains odometer and inertial measurement unit data from the pipe inspection robot.

[0007] The original image sequence is distorted based on the internal geometric parameters of the pipe to obtain a corrected image sequence.

[0008] The corrected image sequence is registered using the odometer and inertial measurement unit data to obtain a panoramic unfolded image of the pipe's inner wall.

[0009] Defect identification is performed based on the panoramic unfolded image.

[0010] Furthermore, the method for obtaining a corrected image sequence by performing distortion correction on the original image sequence based on the internal geometric parameters of the pipe includes: Based on the pipeline axis A pipeline coordinate system is established with the axis and the camera optical center as the origin; based on the transformation relationship between the pipeline coordinate system and the camera coordinate system, the spatial points in the pipeline coordinate system are... Projecting onto the camera's normalized plane yields the projection. , ),in , ; the projection ( , The distorted and normalized coordinates are obtained through the distortion model. , The distortion model is as follows: .

[0011] in, ; , , Radial distortion coefficient; , denoted as the tangential distortion coefficient.

[0012] Obtain pixel coordinates from the original image sequence The distorted normalized coordinates are obtained through inverse transformation of camera intrinsic parameters. , According to the distorted normalized coordinates ( , The distortion-free normalized coordinates are obtained from the distortion model. , According to the distortion-free normalized coordinates ( , The corrected pixel coordinates are obtained through camera intrinsic parameter transformation. The corrected image sequence is obtained based on the corrected pixel coordinates.

[0013] Furthermore, the method for obtaining the radial distortion coefficient and the tangential distortion coefficient includes: Edge detection is performed on the original image sequence to obtain an edge point set; circumferential features of the inner wall of the pipe are extracted based on the edge point set; ellipse fitting is performed on the circumferential features to obtain ellipse parameters; spatial points on the cylindrical surface of the pipe are obtained based on the geometric relationship between the ellipse parameters and the pipe radius; an optimization objective is established based on the spatial points being located on the cylindrical surface of the pipe as a constraint condition, and the formula for the optimization objective is: .

[0014] The constraints are: .

[0015] in, For projection functions; This is the camera intrinsic parameter matrix; This is the distortion coefficient vector; A spatial point on the cylindrical surface of the pipe; Image points on the ellipse.

[0016] The radial distortion coefficient and tangential distortion coefficient are obtained by iteratively solving based on the optimization objective.

[0017] Furthermore, the method for obtaining spatial points on the cylindrical surface of the pipe based on the geometric relationship between the ellipse parameters and the pipe radius includes: Image points are obtained by uniformly sampling on the ellipse. Based on the circumferential features, the axial position of the plane located in space perpendicular to the pipe axis is set as follows: According to the image points Back projection yields a spatial ray; based on the spatial ray and the plane... The intersection of the cylindrical surface of the pipe and the corresponding spatial point is obtained. .

[0018] Furthermore, the method for registering the corrected image sequence based on the odometer and inertial measurement unit data to obtain a panoramic unfolded image of the pipe inner wall includes: The odometer and inertial measurement unit data are aligned to the timestamp of the calibration image using linear interpolation. The formula for the linear interpolation is: .

[0019] in, For odometer readings, For the attitude angle of the inertial measurement unit, and These are the adjacent sampling times of the sensor.

[0020] Odometry displacement difference based on adjacent frame correction images and attitude change of inertial measurement unit Construct an initial pose transformation matrix, the formula of which is: ;in, To change the pitch angle Roll angle variation Heading angle change The rotation matrix formed, It is a translation vector.

[0021] Feature points are extracted from the corrected image sequence; the positions of the feature points in adjacent frames are predicted based on the initial pose transformation matrix; a neighborhood search range is selected based on the predicted positions of the feature points, and feature matching is performed; the initial pose transformation matrix is ​​optimized based on the feature matching results to obtain the optimal pose; the corrected image sequence is mapped onto the cylindrical surface of the pipe based on the optimal pose and unfolded into a planar panoramic image.

[0022] Furthermore, the method for selecting the neighborhood search range based on the predicted location of the feature points and performing feature matching includes: Obtain the distortion-free normalized coordinates of the feature points in the previous frame of the corrected image; assume the feature points are located on the cylindrical surface of the inner wall of the pipe based on the pipe's geometric constraints to obtain the depth value of the feature points; calculate the three-dimensional spatial coordinates of the feature points in the previous frame's camera coordinate system based on the distortion-free normalized coordinates and the depth value; transform the three-dimensional spatial coordinates to the current frame's camera coordinate system based on the initial pose transformation matrix to obtain the transformed three-dimensional spatial coordinates; obtain the predicted position in the current frame of the corrected image based on the transformed three-dimensional spatial coordinates through camera projection. Obtain the covariance matrix of the odometry pose estimation; calculate the covariance matrix of the predicted position in the corrected image plane through error propagation based on the transformed 3D spatial coordinates and the covariance matrix; and calculate the covariance matrix of the covariance matrix of the corrected image plane based on the eigenvalues ​​of the covariance matrix. , The major and minor axes of the search elliptical region are calculated; feature points of the current frame's corrected image are searched within the elliptical region formed by the predicted position as the center, the major axis radius, and the minor axis radius; descriptor distances are calculated based on the feature points in the neighborhood, and the feature point with the smallest descriptor distance is selected as a candidate matching pair; outliers are removed from the candidate matching pairs to obtain the matching feature points.

[0023] Furthermore, the method for optimizing the initial pose transformation matrix based on the feature matching result to obtain the optimal pose includes: Obtain the image coordinates and corresponding spatial points of the matching feature points; calculate the visual reprojection error term based on the image coordinates and spatial points of the matching feature points; obtain the initial pose provided by the odometry; calculate the odometry constraint term based on the pose transformation matrix to be optimized and the initial pose; obtain the attitude rotation matrix measured by the inertial measurement unit; calculate the inertial measurement unit attitude constraint term based on the rotation component in the pose transformation matrix to be optimized and the attitude rotation matrix; construct a joint optimization objective function based on the visual reprojection error term, the odometry constraint term, and the inertial measurement unit attitude constraint term. The number of feature matches is statistically determined based on the feature matching results. The distribution range is calculated based on the spatial distribution of the matched feature points in the image; the distribution range is the ratio of the area of ​​the smallest bounding rectangle formed by the matched feature points to the total area of ​​the image. The visual registration confidence level is calculated based on the number of feature matches and the distribution range. The weights of each term in the joint optimization objective function are adjusted based on the visual registration confidence level. When the visual registration confidence level is not less than a preset threshold, the visual reprojection error term is set as the dominant term in the joint optimization objective function, and the odometry constraint term and the inertial measurement unit attitude constraint term are set as regularization constraint terms. When the visual registration confidence level is less than a preset threshold, the odometry constraint term and the inertial measurement unit attitude constraint term are set as dominant terms in the joint optimization objective function, and the visual reprojection error term is set as an auxiliary term. The optimal pose is obtained by optimizing the adjusted joint optimization objective function.

[0024] Furthermore, the method for mapping the corrected image sequence onto the cylindrical surface of the pipe based on the optimal pose and unfolding it into a planar panoramic image includes: Based on the optimal pose, the pixels in the corrected image sequence are mapped to a cylindrical surface in the pipeline coordinate system to obtain spatial point coordinates; based on the spatial point coordinates... The coordinates of the panoramic image are obtained through unfolding transformation. The formula for the expansion transformation is: ; A panoramic image is generated based on the coordinates of the panoramic image; an overlapping region is obtained based on the mapping position of the corrected images in the panoramic image of adjacent frames; image weights are calculated based on the normalized distance from the pixels in the overlapping region to their respective image centers; the pixel values ​​in the overlapping region are weighted and fused based on the image weights; and a panoramic unfolded image of the inner wall of the pipe is obtained after all the corrected image sequences have undergone the above mapping, unfolding transformation and weighted fusion.

[0025] Furthermore, the method for defect identification based on the panoramic unfolded image includes: Obtain a panoramic unfolded image of the inner wall of the pipe; perform image enhancement on the panoramic unfolded image to obtain an enhanced image; perform defect detection on the enhanced image to obtain candidate defect regions; extract defect features based on the spatial location and geometric shape of the candidate defect regions to obtain defect feature vectors; classify defects based on the defect feature vectors to obtain defect types; obtain defect identification results based on the defect types and the spatial location of the candidate defect regions.

[0026] Based on the same inventive concept, the present invention also provides a pipe inner wall defect identification system based on machine vision, the system comprising: The data acquisition module is used to acquire the original image sequence of the inner wall of the pipeline and obtain the odometer and inertial measurement unit data of the pipeline inspection robot.

[0027] The image distortion correction module is used to perform distortion correction on the original image sequence based on the internal geometric parameters of the pipe to obtain a corrected image sequence.

[0028] The image registration module is used to register the corrected image sequence based on the data from the odometer and the inertial measurement unit to obtain a panoramic unfolded image of the inner wall of the pipe.

[0029] The defect identification module is used to identify defects based on the panoramic unfolded image.

[0030] (3) Beneficial effects Compared with the prior art, the beneficial effects of the present invention are: 1. Camera self-calibration is achieved by utilizing the circumferential features of the inner wall of the pipe, solving the problem that the narrow space inside the pipe makes calibration impossible using a calibration plate.

[0031] 2. By integrating odometer, inertial measurement unit and visual information for joint optimization, and adaptively adjusting the weights according to the visual registration confidence, the registration problem in sparse areas of pipeline features is effectively solved, and the accuracy of pipeline inner wall defect imaging is improved. Attached Figure Description

[0032] Figure 1 This is a flowchart of a method for identifying defects in the inner wall of a pipe based on machine vision, according to Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of the module composition of a machine vision-based pipeline inner wall defect identification system according to Embodiment 2 of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0034] Before providing examples, it is necessary to describe the application scenarios of this invention. It is applicable to the internal wall inspection of cylindrical pipes such as urban drainage networks and industrial pipelines.

[0035] Example 1: As Figure 1 As shown, this embodiment provides a method for identifying defects in the inner wall of a pipe based on machine vision. The method includes: Acquire raw image sequences of the inner wall of the pipe and obtain odometer and inertial measurement unit data from the pipe inspection robot; For example, a pipe inspection robot equipped with a fisheye camera is used to inspect the target pipe. The pipe inspection robot moves along the pipe axis at a constant speed of 0.1 m / s, and the fisheye camera acquires a sequence of raw images at a frequency of 10 frames / second, with an image resolution of 1920×1080 pixels. Simultaneously acquired data also includes odometry data (sampling frequency 20 Hz) and inertial measurement unit data (sampling frequency 100 Hz, including three-axis acceleration and three-axis angular velocity).

[0036] The original image sequence is distorted based on the internal geometric parameters of the pipe to obtain a corrected image sequence. The corrected image sequence is registered based on the odometer and inertial measurement unit data to obtain a panoramic unfolded image of the pipe's inner wall. Defect identification is performed based on the panoramic unfolded image.

[0037] The method for obtaining a corrected image sequence by performing distortion correction on the original image sequence based on the internal geometric parameters of the pipe includes: Based on the pipeline axis A pipeline coordinate system is established with the axis and the camera optical center as the origin; based on the transformation relationship between the pipeline coordinate system and the camera coordinate system, the spatial points in the pipeline coordinate system are... Projecting onto the camera's normalized plane yields the projection. , ),in , ; the projection ( , The distorted and normalized coordinates are obtained through the distortion model. , The distortion model is as follows: .

[0038] in, ; , , Radial distortion coefficient; , denoted as the tangential distortion coefficient.

[0039] Obtain pixel coordinates from the original image sequence The distorted normalized coordinates are obtained through inverse transformation of camera intrinsic parameters. , According to the distorted normalized coordinates ( , The distortion-free normalized coordinates are obtained from the distortion model. , According to the distortion-free normalized coordinates ( , The corrected pixel coordinates are obtained through camera intrinsic parameter transformation. The corrected image sequence is obtained based on the corrected pixel coordinates.

[0040] For example, based on the camera optical center being located at (0,0,0) and the pipe axis being... Establish a pipe coordinate system based on the pipe's inner diameter of 800mm. The pipe axis is then determined to pass through the point (0, -200, 0) and be parallel to the given coordinate system. The shaft, the bottom of the pipe is located =-400mm. Place the point in the pipe coordinate system. Based on the pose provided by the pipeline inspection robot's odometer and inertial measurement unit data, it is transformed into the camera coordinate system at the current shooting moment, and then based on the transformation relationship... , Get the projection ( , ).

[0041] Obtain pixel coordinates from the original image sequence The distorted normalized coordinates are obtained through inverse transformation of camera intrinsic parameters. , The conversion formula is: ; ;in, , The coordinates of the camera's principal point; , This refers to the camera's focal length. In this embodiment, the camera intrinsic parameter matrix... The focal length is [1200,0,960;0,1200,540;0,0,1]. 1200 pixels, principal point coordinates =960 pixels =540 pixels. Select a feature point within the original image. =(1650,890), the distorted normalized coordinates are obtained through inverse transformation of camera intrinsic parameters. =(1650-960) / 1200=0.575, =(890-540) / 1200=0.292. Based on the distorted normalized coordinates ( , The distortion-free normalized coordinates are obtained through the inverse transformation of the distortion model. , The solution is obtained using Newton's iteration method, with the initial value set as (). , The iterative formula is: ,in, The Jacobian matrix is ​​a 2×2 partial derivative matrix of the distortion model with respect to the distortion-free normalized coordinates. The distortion-free normalized coordinates are obtained by solving the inverse transformation of the distortion model using Newton's iteration method for a feature point in the original image. , ) = (0.612, 0.308). Based on the distortion-free normalized coordinates ( , The corrected pixel coordinates are obtained through camera intrinsic parameter transformation. The specific formula is as follows For the aforementioned feature points, the corrected pixel coordinates =1200×0.612+960=1694.4 pixels =1200×0.308+540=909.6 pixels. Based on the corrected pixel coordinates, a corrected image sequence is obtained by generating the corrected image using bilinear interpolation.

[0042] The methods for obtaining the radial distortion coefficient and the tangential distortion coefficient include: Edge detection is performed on the original image sequence to obtain an edge point set; circumferential features of the inner wall of the pipe are extracted based on the edge point set; ellipse fitting is performed on the circumferential features to obtain ellipse parameters; spatial points on the cylindrical surface of the pipe are obtained based on the geometric relationship between the ellipse parameters and the pipe radius; an optimization objective is established based on the spatial points being located on the cylindrical surface of the pipe as a constraint condition, and the formula for the optimization objective is: .

[0043] The constraints are: .

[0044] in, For projection functions; This is the camera intrinsic parameter matrix; This is the distortion coefficient vector; A spatial point on the cylindrical surface of the pipe; Image points on the ellipse.

[0045] The radial distortion coefficient and tangential distortion coefficient are obtained by iteratively solving based on the optimization objective.

[0046] For example, edge detection is performed on the original image sequence to obtain an edge point set. Circumferential features of the pipe's inner wall are extracted based on these edge point sets, including weld rings and interface rings, etc. These circumferential features are then used... The algorithm performs ellipse fitting to obtain ellipse parameters. The number of iterations is set to 1000, and the interior point threshold is set to 2 pixels. The fitted ellipse parameters are: center coordinates (960, 540) pixels, major axis 850 pixels, minor axis 720 pixels, and rotation angle 2.5°. 100 image points are uniformly sampled on the ellipse, and the axial position of the plane containing the circular feature is set. =1000mm (relative to the camera's optical center). For each image point Through camera intrinsic parameter matrix The spatial ray is obtained by back projection. Based on the spatial ray and the plane... The intersection point with the cylindrical surface of the pipe yields a spatial point on the cylindrical surface of the pipe. Based on the spatial point An optimization objective is established using the cylindrical surface of the pipe as a constraint condition. The formula for the optimization objective is: The constraints are: .in, The distortion coefficient vector ; The projection function represents the process of projecting 3D points in the pipeline coordinate system onto image pixel coordinates through the camera intrinsic parameter matrix and distortion coefficient vector. The radial and tangential distortion coefficients are obtained iteratively based on the optimization objective. The algorithm iteratively solves the problem and converges after 50 iterations, yielding the following distortion coefficients: =-0.2847, =0.0943, =-0.0128, =0.0003, =-0.0001.

[0047] The method for obtaining spatial points on the cylindrical surface of the pipe based on the geometric relationship between the ellipse parameters and the pipe radius includes: Image points are obtained by uniformly sampling on the ellipse. Based on the circumferential features, the axial position of the plane located in space perpendicular to the pipe axis is set as follows: According to the image points Back projection yields a spatial ray; based on the spatial ray and the plane... The intersection of the cylindrical surface of the pipe and the corresponding spatial point is obtained. .

[0048] For example, 100 image points are uniformly sampled on the ellipse. The axial position of the plane is set in space perpendicular to the pipe axis based on the circumferential features. Set the axial position of the plane containing the circular feature. =1000mm (relative to the camera optical center). Based on the image point... The spatial ray is obtained through back projection. This is then processed using the camera intrinsic parameter matrix. The space ray is obtained by back projection, and the parametric equation of the space ray is: ,in, For the camera optical center, For image points The ray direction vector; The ray distance parameter and The coordinates of the first image point on the ellipse are... Distortion-free normalized coordinates are obtained through inverse transformation of camera intrinsic parameters. =(1285-960) / 1200=0.271, =(540-540) / 1200=0, the ray direction vector is obtained by back projection. =(0.271;0;1). Based on the spatial ray and plane... The intersection of the cylindrical surface of the pipe and the corresponding spatial point is obtained. Based on the spatial ray and plane of the first image point. The intersection point is (271, 0, 1000) mm. Further, it intersects with the cylindrical surface. Find the intersection to obtain the spatial point =(271,294,1000)mm, where the point satisfies the cylindrical surface constraint: 271 2 +294 2 =73441+86436=159877≈160000=400 2 The error is 0.08%.

[0049] The method for registering the corrected image sequence based on the odometer and inertial measurement unit data to obtain a panoramic unfolded image of the pipe inner wall includes: The odometer and inertial measurement unit data are aligned to the timestamp of the calibration image using linear interpolation. The formula for the linear interpolation is: .

[0050] in, For odometer readings, For the attitude angle of the inertial measurement unit, and These are the adjacent sampling times of the sensor.

[0051] Odometry displacement difference based on adjacent frame correction images and attitude change of inertial measurement unit Construct an initial pose transformation matrix, the formula of which is: ;in, To change the pitch angle Roll angle variation Heading angle change The rotation matrix formed, It is a translation vector.

[0052] Feature points are extracted from the corrected image sequence; the positions of the feature points in adjacent frames are predicted based on the initial pose transformation matrix; a neighborhood search range is selected based on the predicted positions of the feature points, and feature matching is performed; the initial pose transformation matrix is ​​optimized based on the feature matching results to obtain the optimal pose; the corrected image sequence is mapped onto the cylindrical surface of the pipe based on the optimal pose and unfolded into a planar panoramic image.

[0053] For example, odometer and inertial measurement unit data are aligned to the timestamp of the calibration image using linear interpolation. The formula for the linear interpolation is: .

[0054] Among them, the timestamp is =0.1 Seconds are used for linear interpolation alignment between adjacent odometer sampling times. Odometer displacement difference is corrected based on adjacent frame images. and attitude change of inertial measurement unit Construct the initial pose transformation matrix. In this embodiment, the pipeline inspection robot travels along a horizontal straight pipe section with minimal attitude change and only slight pitch angle change. =0.5°, roll angle change =0.3°, heading angle change =0.1°, axial displacement =10mm to obtain the initial pose transformation matrix .

[0055] Use for correcting image sequences A feature detector extracts feature points. 500 feature points are extracted from each frame of the corrected image, with a detection threshold set to 20 and a pyramid layer count of 8. The positions of these feature points in adjacent frames are predicted based on the initial pose transformation matrix. The position of the feature points in adjacent frames is predicted based on the pipeline geometric constraints. The feature points in the frame are located on the cylindrical surface of the inner wall of the pipe. The corresponding depth values ​​and three-dimensional spatial coordinates are calculated, and then transformed to the next frame using the initial pose transformation matrix. The predicted positions of feature points are obtained by projecting the camera coordinates of frame +1 onto the target frame. A neighborhood search range is selected based on the predicted positions, and feature matching is performed. The search is then conducted within the elliptical neighborhood of the predicted positions. The descriptor distance (Hamming distance) is calculated from the feature points of the +1 frame. The feature point with the smallest distance that is less than the threshold of 50 is selected as the candidate matching pair. The initial pose transformation matrix is ​​optimized based on the feature matching results to obtain the optimal pose. After eliminating outliers, the algorithm constructs a joint optimization objective function for pose optimization. In this embodiment, the first... Frame and the The number of successfully matched feature point pairs between +1 frames is =327 pairs. The corrected image sequence is mapped onto the cylindrical surface of the pipe according to the optimal pose and unfolded into a planar panoramic image. The unfolded panoramic image has a size of 50000×2513 pixels (width×height). Adjacent frame corrected images have overlapping areas in the panoramic image, requiring fusion processing to eliminate stitching gaps. Image fusion processing is performed on the overlapping areas to obtain the final unfolded panoramic image.

[0056] The method for selecting the neighborhood search range based on the predicted location of feature points and performing feature matching includes: Obtain the distortion-free normalized coordinates of the feature points in the previous frame of the corrected image; assume the feature points are located on the cylindrical surface of the inner wall of the pipe based on the pipe's geometric constraints to obtain the depth value of the feature points; calculate the three-dimensional spatial coordinates of the feature points in the previous frame's camera coordinate system based on the distortion-free normalized coordinates and the depth value; transform the three-dimensional spatial coordinates to the current frame's camera coordinate system based on the initial pose transformation matrix to obtain the transformed three-dimensional spatial coordinates; obtain the predicted position in the current frame of the corrected image through camera projection based on the transformed three-dimensional spatial coordinates.

[0057] Obtain the covariance matrix of the odometry pose estimation; calculate the covariance matrix of the predicted position in the corrected image plane through error propagation based on the transformed 3D spatial coordinates and the covariance matrix; and calculate the covariance matrix of the covariance matrix of the corrected image plane based on the eigenvalues ​​of the covariance matrix. , The major and minor axes of the search elliptical region are calculated; feature points of the current frame's corrected image are searched within the elliptical region formed by the predicted position as the center, the major axis radius, and the minor axis radius; descriptor distances are calculated based on the feature points in the neighborhood, and the feature point with the smallest descriptor distance is selected as a candidate matching pair; outliers are removed from the candidate matching pairs to obtain the matching feature points.

[0058] For example, obtain the distortion-free normalized coordinates of feature points in the previous frame of the corrected image. , The depth value of the feature point is obtained by assuming that the feature point is located on the cylindrical surface of the inner wall of the pipe based on the pipe's geometric constraints. The formula for calculating the depth value is: ;in, =400mm is the pipe radius. The distortion-free normalized coordinates of a feature point in the frame are (0.25, 0.15), and the depth value is 1372 mm according to the depth formula. Based on the distortion-free normalized coordinates (... , ) and depth value The three-dimensional spatial coordinates of the feature point in the camera coordinate system of the previous frame were calculated. , The three-dimensional spatial coordinates are calculated based on the feature points. =(343,206,1372)mm. The three-dimensional spatial coordinates are transformed to the current frame camera coordinate system based on the initial pose transformation matrix to obtain the transformed three-dimensional spatial coordinates. The transformation formula is: The feature points are obtained after transformation by the initial pose transformation matrix. =(343,206,1382)mm. The predicted position in the current frame's corrected image is obtained by projecting the transformed three-dimensional spatial coordinates onto the camera. The calculation formula is: ; ;in, , and Transformed three-dimensional spatial coordinates The three coordinate components are calculated. The projection onto the first... +1 frame of corrected image to obtain the predicted location =(1257,719)mm.

[0059] Obtain the covariance matrix of odometry pose estimation ,in, = =0.01mm; =0.05mm; the covariance matrix is ​​determined based on the odometer's measurement data and historical error statistics. The covariance matrix of the predicted position on the corrected image plane is calculated through error propagation based on the transformed three-dimensional spatial coordinates and the covariance matrix. The covariance matrix The calculation formula is: .in, The covariance matrix is ​​a 2×6 partial derivative matrix of the projection function with respect to the pose parameters, which are the rotation and translation components of the initial pose transformation matrix. Eigenvalues ​​are obtained by performing eigenvalue decomposition. For 25, The value is 7.11. Based on the eigenvalues ​​of the covariance matrix... , Obtain the major axis radius of the search elliptical region =15 pixels, minor axis radius =8 pixels, where, , Feature points of the current frame's corrected image are searched within an elliptical region centered on the predicted location and bounded by its major and minor axes. Descriptor distances are calculated based on the neighboring feature points, and the feature point with the smallest descriptor distance is selected as the candidate matching pair. Three candidate feature points with coordinates (1258, 720), (1253, 716), and (1265, 725) are found within the elliptical region. The descriptor distance is calculated... The Hamming distances of the descriptors are 23, 47, and 52, respectively. The feature point with the smallest distance (1258, 720) is selected as the matching point, and the deviation between the matching point and the predicted position is... The pixels are within a reasonable range. After repeating the above ellipse search process for all feature points, we obtain... =327 candidate matching pairs. Using The algorithm eliminates false matches by randomly selecting 8 pairs of points from the 327 candidate matching pairs, calculating the fundamental matrix, and counting the number of interior points. After 1000 iterations, the model with the most interior points is selected as the optimal solution, and the final matching feature points are obtained after eliminating outliers.

[0060] The method for optimizing the initial pose transformation matrix based on the feature matching result to obtain the optimal pose includes: Obtain the image coordinates and corresponding spatial points of the matching feature points; calculate the visual reprojection error term based on the image coordinates and spatial points of the matching feature points; obtain the initial pose provided by the odometer; calculate the odometer constraint term based on the pose transformation matrix to be optimized and the initial pose; obtain the attitude rotation matrix measured by the inertial measurement unit; calculate the inertial measurement unit attitude constraint term based on the rotation component in the pose transformation matrix to be optimized and the attitude rotation matrix; construct a joint optimization objective function based on the visual reprojection error term, the odometer constraint term, and the inertial measurement unit attitude constraint term.

[0061] The number of feature matches is statistically determined based on the feature matching results. The distribution range is calculated based on the spatial distribution of the matched feature points in the image; the distribution range is the ratio of the area of ​​the smallest bounding rectangle formed by the matched feature points to the total area of ​​the image. The visual registration confidence score is calculated based on the number of feature matches and the distribution range. The weights of each term in the joint optimization objective function are adjusted according to the visual registration confidence score. When the visual registration confidence score is not less than a preset threshold, the visual reprojection error term is set as the dominant term in the joint optimization objective function, and the odometry constraint term and the inertial measurement unit attitude constraint term are set as regularization constraint terms. When the visual registration confidence score is less than a preset threshold, the odometry constraint term and the inertial measurement unit attitude constraint term are set as dominant terms in the joint optimization objective function, and the visual reprojection error term is set as an auxiliary term. The optimal pose is obtained by optimizing the adjusted joint optimization objective function.

[0062] For example, obtaining the image coordinates of matching feature points. and corresponding spatial points Based on the image coordinates of the matched feature points and spatial points The visual reprojection error term is calculated. The calculation formula is: .

[0063] in, =327 represents the number of matched feature point pairs; For the first Index of matching feature points; For the first Image coordinates in frame +1 For the first The coordinates of the corresponding spatial point in the frame, Let be the pose transformation matrix to be optimized. This is the projection function.

[0064] Obtain the initial pose provided by the odometry. Based on the pose transformation matrix to be optimized With the initial pose The odometer constraint term is calculated, and the odometer constraint term... The calculation formula is: .

[0065] in, express Norm.

[0066] Obtain the attitude rotation matrix measured by the inertial measurement unit. Based on the pose transformation matrix to be optimized Rotational components in With the attitude rotation matrix The attitude constraint terms of the inertial measurement unit were calculated. The attitude constraint term of the inertial measurement unit The calculation formula is:

[0067] A joint optimization objective function is constructed based on the visual reprojection error term, the odometry constraint term, and the inertial measurement unit attitude constraint term:

[0068] The number of feature matches is statistically determined based on the feature matching results; the distribution range is calculated based on the spatial distribution of the matched feature points in the image, where the distribution range is the ratio of the area of ​​the smallest bounding rectangle formed by the matched feature points to the total area of ​​the image; and the visual registration confidence level is calculated based on the number of feature matches and the distribution range. The formula for calculating the visual registration confidence level is as follows: .in, =327 represents the number of feature points that were successfully matched. =500 is a preset reference value for the number of features. =0.78 is the ratio of the area of ​​the smallest bounding rectangle formed by the matched feature points to the total area of ​​the image. =1 is a preset distribution range reference value. Based on the visual registration confidence level... Adjust the weight relationships of each term in the joint optimization objective function; when When the visual information is ≥0.5 of the preset threshold, the visual information is reliable, and the weight is set to... =0.8, =0.15, =0.05. Specifically, when When the preset threshold is 0.5, visual information is unreliable; therefore, the weight is set to... =0.2, =0.6, =0.2. The optimal pose is obtained by optimizing the solution based on the adjusted joint optimization objective function. Using... The algorithm is optimized and iterated until convergence is obtained to obtain the optimal pose transformation matrix.

[0069] The method for mapping the corrected image sequence onto the cylindrical surface of the pipe based on the optimal pose and unfolding it into a planar panoramic image includes: Based on the optimal pose, the pixels in the corrected image sequence are mapped to a cylindrical surface in the pipeline coordinate system to obtain spatial point coordinates; based on the spatial point coordinates... The coordinates of the panoramic image are obtained through unfolding transformation. The formula for the expansion transformation is: ; A panoramic image is generated based on the coordinates of the panoramic image; an overlapping region is obtained based on the mapping position of the corrected images in the panoramic image of adjacent frames; image weights are calculated based on the normalized distance from the pixels in the overlapping region to their respective image centers; the pixel values ​​in the overlapping region are weighted and fused based on the image weights; and a panoramic unfolded image of the inner wall of the pipe is obtained after all the corrected image sequences have undergone the above mapping, unfolding transformation and weighted fusion.

[0070] For example, based on the corrected image sequence of the first... Each pixel of the frame image Through camera intrinsic parameter matrix The inverse transformation yields distortion-free normalized coordinates; based on the pipe geometry constraints, assuming the pixel is located on the cylindrical surface of the inner wall of the pipe, and combined with the optimal pose transformation, the three-dimensional spatial coordinates of the pixel in the pipe coordinate system are calculated. Based on the coordinates of the spatial point The coordinates of the panoramic image are obtained through unfolding transformation. The expansion transformation transforms the three-dimensional points in the cylindrical coordinate system of the pipe. Mapping to 2D panoramic coordinates The formula for the expansion transformation is: ; The initial size of the unfolded panoramic image is 50000 × 2513 pixels (width × height). The overlapping area is determined based on the mapping positions of adjacent frame images in the panoramic image. Adjacent frame images partially overlap in the panoramic image, with the overlap width typically being 20%-30% of the width of a single frame. In this embodiment, the width of each frame image mapped in the panoramic image is approximately 100mm, and the overlap width between adjacent frames is approximately 25mm. For each pixel within the overlapping area, the normalized distance from the pixel to its respective image center is used... Calculate weights The weight calculation formula is as follows: ;in, The weight decay parameter and =0.3. The pixel values ​​of the overlapping region are weighted and fused according to the weight, and the fusion formula is: .in, and These are the original grayscale values ​​of pixels in two adjacent frames of the image; The current image frame is indexed. This method enables a smooth transition from one frame to another in the overlapping area, eliminating visible stitching gaps. After processing all the corrected image sequences through the above mapping, unfolding transformation, and fusion, a panoramic unfolded image of the pipe's inner wall is obtained.

[0071] The method for defect identification based on the panoramic unfolded image includes: Obtain a panoramic unfolded image of the inner wall of the pipe; perform image enhancement on the panoramic unfolded image to obtain an enhanced image; perform defect detection on the enhanced image to obtain candidate defect regions; extract defect features based on the spatial location and geometric shape of the candidate defect regions to obtain defect feature vectors; classify defects based on the defect feature vectors to obtain defect types; obtain defect identification results based on the defect types and the spatial location of the candidate defect regions.

[0072] For example, a panoramic view of the pipe's inner wall is obtained. Histogram equalization is used to enhance image contrast, and Gaussian filtering is applied to remove noise and improve the accuracy of subsequent detection. The enhanced image is processed to identify and extract candidate defect regions, including longitudinal cracks, transverse cracks, damage, and leakage. Geometric features (area, aspect ratio, etc.), texture features (gray-level co-occurrence matrix features), and morphological features (principal orientation angle, skeleton length, etc.) are extracted for each candidate region to form a multi-dimensional feature vector. The extracted feature vectors are classified, categorizing defects into cracks (longitudinal / transverse), damage, leakage, and interface misalignment. The position coordinates of the defect in the panoramic image are used to calculate its three-dimensional spatial location in the pipe, and the axial distance, circumferential angle, and radial position of the defect are marked. A defect detection report is generated, including defect type statistics, defect severity assessment, defect spatial distribution map, and maintenance recommendations.

[0073] Example 2: Based on the same inventive concept, such as Figure 2 As shown, this embodiment also provides a machine vision-based pipe inner wall defect identification system, including: The data acquisition module is used to acquire the original image sequence of the inner wall of the pipeline and obtain the odometer and inertial measurement unit data of the pipeline inspection robot.

[0074] The image distortion correction module is used to perform distortion correction on the original image sequence based on the internal geometric parameters of the pipe to obtain a corrected image sequence.

[0075] The image registration module is used to register the corrected image sequence based on the data from the odometer and the inertial measurement unit to obtain a panoramic unfolded image of the inner wall of the pipe.

[0076] The defect identification module is used to identify defects based on the panoramic unfolded image.

[0077] It should be noted that the specific ways in which each module performs operations in the system described in the above embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0078] Finally, it should be noted that although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying defects in the inner wall of a pipe based on machine vision, characterized in that, The method includes: Acquire raw image sequences of the inner wall of the pipe and obtain odometer and inertial measurement unit data from the pipe inspection robot; The original image sequence is distorted based on the internal geometric parameters of the pipe to obtain a corrected image sequence. The corrected image sequence is registered based on the odometer and inertial measurement unit data to obtain a panoramic unfolded image of the pipe's inner wall. Defect identification is performed based on the panoramic unfolded image.

2. The method for identifying defects in the inner wall of a pipe based on machine vision according to claim 1, characterized in that, The method for obtaining a corrected image sequence by performing distortion correction on the original image sequence based on the internal geometric parameters of the pipe includes: Based on the pipeline axis A pipeline coordinate system is established with the axis and the camera optical center as the origin; based on the transformation relationship between the pipeline coordinate system and the camera coordinate system, the spatial points in the pipeline coordinate system are... Projecting onto the camera's normalized plane yields the projection. , ),in , ; the projection ( , The distorted and normalized coordinates are obtained through the distortion model. , The distortion model is as follows: ； in, ; , , Radial distortion coefficient; , The tangential distortion coefficient; Obtain pixel coordinates from the original image sequence The distorted normalized coordinates are obtained through inverse transformation of camera intrinsic parameters. , According to the distorted normalized coordinates ( , The distortion-free normalized coordinates are obtained from the distortion model. , According to the distortion-free normalized coordinates ( , The corrected pixel coordinates are obtained through camera intrinsic parameter transformation. The corrected image sequence is obtained based on the corrected pixel coordinates.

3. The method for identifying defects in the inner wall of a pipe based on machine vision according to claim 2, characterized in that, The methods for obtaining the radial distortion coefficient and the tangential distortion coefficient include: Edge detection is performed on the original image sequence to obtain an edge point set; circumferential features of the inner wall of the pipe are extracted based on the edge point set; ellipse fitting is performed on the circumferential features to obtain ellipse parameters; spatial points on the cylindrical surface of the pipe are obtained based on the geometric relationship between the ellipse parameters and the pipe radius; an optimization objective is established based on the spatial points being located on the cylindrical surface of the pipe as a constraint condition, and the formula for the optimization objective is: ； The constraints are: ; in, For projection functions; This is the camera intrinsic parameter matrix; This is the distortion coefficient vector; A spatial point on the cylindrical surface of the pipe; Image points on the ellipse; The radial distortion coefficient and tangential distortion coefficient are obtained by iteratively solving based on the optimization objective.

4. The method for identifying defects in the inner wall of a pipe based on machine vision according to claim 3, characterized in that, The method for obtaining spatial points on the cylindrical surface of the pipe based on the geometric relationship between the ellipse parameters and the pipe radius includes: Image points are obtained by uniformly sampling on the ellipse. Based on the circumferential features, the axial position of the plane located in space perpendicular to the pipe axis is set as follows: According to the image points Back projection yields a spatial ray; based on the spatial ray and the plane... The intersection of the cylindrical surface of the pipe and the corresponding spatial point is obtained. .

5. The method for identifying defects in the inner wall of a pipe based on machine vision according to claim 4, characterized in that, The method for registering the corrected image sequence based on the odometer and inertial measurement unit data to obtain a panoramic unfolded image of the pipe inner wall includes: The odometer and inertial measurement unit data are aligned to the timestamp of the calibration image using linear interpolation. The formula for the linear interpolation is: ； in, For odometer readings, For the attitude angle of the inertial measurement unit, and These are the adjacent sampling times of the sensor; Odometry displacement difference based on adjacent frame correction images and attitude change of inertial measurement unit Construct an initial pose transformation matrix, the formula of which is: ;in, To change the pitch angle Roll angle variation Heading angle change The rotation matrix formed, It is a translation vector; Feature points are extracted from the corrected image sequence; the positions of the feature points in adjacent frames are predicted based on the initial pose transformation matrix; a neighborhood search range is selected based on the predicted positions of the feature points, and feature matching is performed; the initial pose transformation matrix is ​​optimized based on the feature matching results to obtain the optimal pose; the corrected image sequence is mapped onto the cylindrical surface of the pipe based on the optimal pose and unfolded into a planar panoramic image.

6. The method for identifying defects in the inner wall of a pipe based on machine vision according to claim 5, characterized in that, The method for selecting the neighborhood search range based on the predicted location of feature points and performing feature matching includes: Obtain the distortion-free normalized coordinates of the feature points in the previous frame of the corrected image; assume the feature points are located on the cylindrical surface of the inner wall of the pipe based on the pipe's geometric constraints to obtain the depth value of the feature points; calculate the three-dimensional spatial coordinates of the feature points in the previous frame's camera coordinate system based on the distortion-free normalized coordinates and the depth value; transform the three-dimensional spatial coordinates to the current frame's camera coordinate system based on the initial pose transformation matrix to obtain the transformed three-dimensional spatial coordinates; obtain the predicted position in the current frame of the corrected image based on the transformed three-dimensional spatial coordinates through camera projection. Obtain the covariance matrix of the odometry pose estimation; calculate the covariance matrix of the predicted position in the corrected image plane through error propagation based on the transformed 3D spatial coordinates and the covariance matrix; and calculate the covariance matrix of the covariance matrix of the corrected image plane based on the eigenvalues ​​of the covariance matrix. , The major and minor axes of the search elliptical region are calculated; feature points of the current frame's corrected image are searched within the elliptical region formed by the predicted position as the center, the major axis radius, and the minor axis radius; descriptor distances are calculated based on the feature points in the neighborhood, and the feature point with the smallest descriptor distance is selected as a candidate matching pair; outliers are removed from the candidate matching pairs to obtain the matching feature points.

7. The method for identifying defects in the inner wall of a pipe based on machine vision according to claim 6, characterized in that, The method for optimizing the initial pose transformation matrix based on the feature matching result to obtain the optimal pose includes: Obtain the image coordinates and corresponding spatial points of the matching feature points; calculate the visual reprojection error term based on the image coordinates and spatial points of the matching feature points; obtain the initial pose provided by the odometry; calculate the odometry constraint term based on the pose transformation matrix to be optimized and the initial pose; obtain the attitude rotation matrix measured by the inertial measurement unit; calculate the inertial measurement unit attitude constraint term based on the rotation component in the pose transformation matrix to be optimized and the attitude rotation matrix; construct a joint optimization objective function based on the visual reprojection error term, the odometry constraint term, and the inertial measurement unit attitude constraint term. The number of feature matches is statistically determined based on the feature matching results. The distribution range is calculated based on the spatial distribution of the matched feature points in the image; the distribution range is the ratio of the area of ​​the smallest bounding rectangle formed by the matched feature points to the total area of ​​the image. The visual registration confidence level is calculated based on the number of feature matches and the distribution range. The weights of each term in the joint optimization objective function are adjusted based on the visual registration confidence level. When the visual registration confidence level is not less than a preset threshold, the visual reprojection error term is set as the dominant term in the joint optimization objective function, and the odometry constraint term and the inertial measurement unit attitude constraint term are set as regularization constraint terms. When the visual registration confidence level is less than a preset threshold, the odometry constraint term and the inertial measurement unit attitude constraint term are set as dominant terms in the joint optimization objective function, and the visual reprojection error term is set as an auxiliary term. The optimal pose is obtained by optimizing the adjusted joint optimization objective function.

8. The method for identifying defects in the inner wall of a pipe based on machine vision according to claim 7, characterized in that, The method for mapping the corrected image sequence onto the cylindrical surface of the pipe based on the optimal pose and unfolding it into a planar panoramic image includes: Based on the optimal pose, the pixels in the corrected image sequence are mapped to a cylindrical surface in the pipeline coordinate system to obtain spatial point coordinates; based on the spatial point coordinates... The coordinates of the panoramic image are obtained through unfolding transformation. The formula for the expansion transformation is: ; A panoramic image is generated based on the coordinates of the panoramic image; an overlapping region is obtained based on the mapping position of the corrected images in the panoramic image of adjacent frames; image weights are calculated based on the normalized distance from the pixels in the overlapping region to their respective image centers; the pixel values ​​in the overlapping region are weighted and fused based on the image weights; and a panoramic unfolded image of the inner wall of the pipe is obtained after all the corrected image sequences have undergone the above mapping, unfolding transformation and weighted fusion.

9. A method for identifying defects in the inner wall of a pipe based on machine vision according to claim 8, characterized in that, The method for defect identification based on the panoramic unfolded image includes: Obtain a panoramic unfolded image of the inner wall of the pipe; perform image enhancement on the panoramic unfolded image to obtain an enhanced image; perform defect detection on the enhanced image to obtain candidate defect regions; extract defect features based on the spatial location and geometric shape of the candidate defect regions to obtain defect feature vectors; classify defects based on the defect feature vectors to obtain defect types; obtain defect identification results based on the defect types and the spatial location of the candidate defect regions.

10. A machine vision-based pipe inner wall defect identification system, characterized in that, The system includes: The data acquisition module is used to acquire the original image sequence of the inner wall of the pipeline and obtain the odometer and inertial measurement unit data of the pipeline inspection robot. An image distortion correction module is used to perform distortion correction on the original image sequence based on the internal geometric parameters of the pipe to obtain a corrected image sequence; The image registration module is used to register the corrected image sequence based on the data from the odometer and the inertial measurement unit to obtain a panoramic unfolded image of the inner wall of the pipe. The defect identification module is used to identify defects based on the panoramic unfolded image.

Images