An edge contour-based straight pipe spatial pose visual tracking device and method

Abstract

The present application belongs to the technical field of pipeline pose measurement, and specifically discloses a straight pipeline space pose visual tracking device and method based on edge contour, which uses a camera to collect multiple planar target images under different angles, calibrates the camera to obtain an intrinsic matrix and distortion coefficients of the camera; uses the camera to collect an equal-diameter straight pipeline image with a known radius to perform edge contour feature extraction to obtain contour feature points; corrects the feature points for distortion; and performs linear fitting on the corrected non-distortion contour feature points to obtain two contour lines of the equal-diameter straight pipeline, and calculates the axis space pose according to the constraint relationship between the two contour lines and the axis of the equal-diameter straight pipeline. The present application can obtain the axis pose of the equal-diameter straight pipeline through linear calculation without distortion, has the advantages of simple hardware structure, convenient operation, low cost, etc., can quickly determine the pose of the equal-diameter straight pipeline, and realizes the pose tracking of the equal-diameter straight pipeline.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pipeline pose measurement, and in particular to a straight pipeline space pose visual tracking device and method based on edge contour. BACKGROUND

[0002] As one of the indispensable components in the pipeline system, the equal-diameter straight pipeline is widely used in the transmission system of liquid or gas in many industrial fields such as petroleum, natural gas, and chemical industry, and is an important infrastructure to ensure the continuity of industrial production and the stability of the system. Precise butt joint of the equal-diameter straight pipeline during assembly and laying can improve the safety and reliability of the pipeline system, avoid safety accidents such as leakage at the connection, accelerated corrosion, and pipeline rupture caused by inaccurate butt joint of the pipeline, and provide strong protection for the safe operation of the pipeline system.

[0003] At present, the equal-diameter straight pipeline can be assembled and laid by using the positioning tool method or the pose visual tracking method. However, the assembly of the equal-diameter straight pipeline by using the positioning tool method requires the use of complex tooling equipment such as clamps, and manual adjustment and monitoring, which may cause large errors in positioning. The pose visual tracking can provide high-precision alignment information to ensure accurate butt joint of the pipeline during installation and reduce errors. Moreover, the alignment of the pipeline is monitored in real time, and deviations are corrected in time to improve work efficiency.

[0004] The invention patent with the application number CN202210182068.0 "Pipeline pose measurement device and method" proposes a method for detecting the pose of the pipeline. The method first rotates the laser around the circumference by any angle to project a laser point on the inner wall of the pipeline, then performs ellipse fitting on the laser point to determine the axis vector of the ellipse, and finally rotates the coordinate system established at the center of the ellipse to obtain the pipeline axis coordinate system, thereby completing the measurement of the pipeline pose.

[0005] The invention patent with the application number CN202110217140.4 "Cylinder pose measurement method and device" proposes a method for calculating the attitude of a cylinder. The method first obtains the relevant parameters of the cylinder through the line laser displacement sensor arranged around the cylinder, then calculates the center positions of two height measurement sections of the cylinder using a cylinder pose measurement model. The line connecting the center positions is the attitude direction of the cylinder, and the ratio of the projection deviation to the vertical movement height of the line laser displacement sensor is the attitude inclination of the cylinder, thereby determining the attitude of the cylinder.

[0006] Patent application number CN202410433127.6, "A Pipeline Centerline Identification Method, System, Computer Device, and Storage Medium," proposes a method for determining a pipeline centerline. This method first uses laser sensors, such as lidar, to directly acquire 3D point cloud data of the pipeline. The method then extracts the centerline from this 3D point cloud data to determine the pipeline's trajectory.

[0007] However, the above three solutions all require the use of multiple sensor devices, which makes the device or system structure more complex, the posture solution process more cumbersome, and the cost relatively high; and the speed of determining the posture of straight pipes with equal diameters is also slow. Summary of the Invention

[0008] In order to solve the above technical problems, the present invention provides a device and method for visually tracking the spatial posture of a straight pipe based on edge contours.

[0009] To achieve the above object, the present invention is implemented according to the following technical solutions:

[0010] The first object of the present invention is to provide a straight pipe spatial posture visual tracking method based on edge contour, comprising the following steps:

[0011] S1. Lay a two-dimensional plane target horizontally on an operating platform, and fix a camera obliquely above the two-dimensional plane target. Place the two-dimensional plane target within the camera's field of view and adjust its position. Use the camera to capture multiple images of the two-dimensional plane target at different positions. Use Zhang's calibration method to calibrate the camera based on the multiple two-dimensional plane target images captured by the camera to obtain the camera's intrinsic parameter matrix and distortion coefficient.

[0012] S2. Place a straight pipe of equal diameter with a known radius horizontally within the camera's field of view using a bracket, and use the camera to capture an image of the straight pipe of equal diameter; extract edge contour features from the image of the straight pipe of equal diameter to obtain distorted contour feature points;

[0013] S3, combining the camera's intrinsic parameter matrix and distortion coefficient to perform distortion correction on the distorted contour feature points to obtain undistorted contour feature points;

[0014] S4. Perform linear fitting on the undistorted contour feature points to obtain two contour lines of straight pipes with equal diameters;

[0015] S5. Combining the two contour lines of the straight pipe, the radius of the straight pipe, and the intrinsic parameter matrix of the camera, the spatial pose of the straight pipe is calculated according to the perspective projection model of the straight pipe.

[0016] Furthermore, in step S2, the canny edge detection algorithm is used to extract edge contour features from the image of straight pipes of equal diameter.

[0017] Further, the step S3, the calculation formula of distortion correction is shown as formula (1):

[0018]

[0019] Wherein: and (u, v) respectively represent the pixel coordinates of the profile feature points with distortion and without distortion; (x, y) represents the normalized image coordinates without distortion; (k1, k2) represents the radial distortion coefficient of the camera; (u0, v0) represents the principal point coordinates in the intrinsic matrix of the camera.

[0020] Further, the step S4 includes:

[0021] S41, using RANSAC algorithm to screen the profile feature points without distortion to obtain the feature point sets {(u' g ,v' g )|g∈1,...,G} and {(u' m ,v' m )|m∈1,...,M} on the two profiles of the equal-diameter straight pipe, wherein: (u' g ,v' g ) and (u' m ,v' m ) respectively represent the pixel coordinates of the feature points on the two profiles;

[0022] S42, based on the least square method, the feature point sets are respectively fitted to obtain the two profile lines l1 and l2 of the equal-diameter straight pipe, and the equation of the fitting straight line is shown as formula (2):

[0023]

[0024] Wherein: a', b', c' and a'', b'', c'' are respectively the equation coefficients of the two fitting straight lines; the solving process of a', b', c' and a'', b'', c'' is as follows:

[0025] The equation group composed of the homogeneous coordinates of the feature points on the two profiles as the coefficient matrix is shown as formula (3):

[0026]

[0027] For formula (3), the characteristic vector corresponding to the minimum eigenvalue of the coefficient matrix is the least square solution of the equation group; or the SVD decomposition is performed on the coefficient matrix, and then the vector corresponding to the minimum singular value in the right singular matrix is taken as the least square solution;

[0028] The obtained least square solution is brought into formula (2) to obtain the two profile lines l1 and l2 of the equal-diameter straight pipe.

[0029] Furthermore, the step S5 includes:

[0030] S51. Combining the camera's intrinsic parameter matrix and the two contour lines l1 and l2 of the straight pipe with equal diameter, the back-projection planes corresponding to the two contour lines are:

[0031]

[0032] Where: K represents the camera intrinsic parameter matrix, π1 and π2 represent the back-projection planes of the contour lines corresponding to l1 and l2 respectively;

[0033] S52, according to the obtained back-projection planes π1 and π2, obtain their normal vectors n1 and n2, and calculate the bisector π of the back-projection planes π1 and π2 m The normal vector is:

[0034]

[0035] Where: n m is the normal vector of the bisector of the two back-projection planes, ||n|| represents the modulus of the vector, and the angle and intersection vector of the back-projection planes π1 and π2 are calculated as follows:

[0036]

[0037] Where: α is the angle between the two back-projection planes, v L represents the intersection vector of the back-projection planes π1 and π2;

[0038] S53, according to the obtained intersection vector v L and the bisecting plane normal vector n m Calculate the center P of the cross section from the origin O to the support plane passing through the origin C The direction vector is:

[0039]

[0040] in: means OP C The direction vector of the straight pipe of equal diameter is combined with the radius r of the straight pipe and the angle α between the two back-projection planes to calculate the origin O and P. C The distance between points is:

[0041]

[0042] in: Indicates point O to P C The distance between the points, according to OP C Direction vector and point O and P C Distance of points get P C three-dimensional coordinates of the points:

[0043]

[0044] Since the radius of the equal-diameter straight pipe is known, the spatial pose of the equal-diameter straight pipe is determined by P C three-dimensional coordinates of the points and a direction vector v L of the axis.

[0045] A second object of the present application is to provide an edge contour-based straight pipe spatial pose visual tracking device, comprising:

[0046] an operation platform for laying a two-dimensional planar target;

[0047] a two-dimensional planar target, which is movable to be laid on the operation platform to adjust a plurality of different positions within a camera field of view range;

[0048] a camera, which is fixed on the two-dimensional planar target obliquely above by a rack, and the two-dimensional planar target is always located within the camera field of view range during a camera calibration process;

[0049] a support, which is arranged on the operation platform at both sides of the two-dimensional planar target;

[0050] an equal-diameter straight pipe, which is arranged on the support and located within the camera field of view range;

[0051] a processing system, which is connected with the camera and executes the above-mentioned edge contour-based straight pipe spatial pose visual tracking method.

[0052] Preferably, the two-dimensional planar target is a two-dimensional planar checkerboard.

[0053] Compared with the prior art, the present application firstly uses a camera to collect a plurality of planar target images at different angles, calibrates the camera to obtain an intrinsic matrix and distortion coefficients of the camera; secondly, uses the camera to collect an equal-diameter straight pipe image with a known radius to perform edge contour feature extraction to obtain contour feature points; since there is lens distortion, the feature points need to be distortion-corrected; then, the distortion-corrected non-distortion contour feature points are respectively subjected to straight line fitting to obtain two contour lines of the equal-diameter straight pipe; finally, the spatial pose of the axis is solved according to the constraint relationship between the two contour lines and the axis of the equal-diameter straight pipe; the present application can obtain the axis pose of the equal-diameter straight pipe through linear calculation without distortion, has the advantages of simple hardware structure, convenient operation, low cost, etc., can quickly determine the pose of the equal-diameter straight pipe, and realize the pose tracking of the equal-diameter straight pipe; reduces the butt joint error of the equal-diameter straight pipe in actual assembly and laying work, and improves the efficiency of pipe system assembly and laying. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 This is a structural layout diagram of the camera calibration process of the edge contour-based straight pipe spatial posture visual tracking device of the present invention.

[0055] Figure 2 This is a schematic structural diagram of a two-dimensional planar target of the present invention.

[0056] Figure 3 This is a structural layout diagram of the process of collecting images of straight pipes of equal diameter by the straight pipe spatial posture visual tracking device based on edge contours of the present invention.

[0057] Figure 4 This is a flow chart of the edge contour-based straight pipe spatial posture visual tracking method of the present invention.

[0058] Figure 5 It is the perspective projection model of a straight pipe with equal diameter.

[0059] Figure 6 These are the two contour lines of the straight pipe with equal diameter obtained by fitting.

[0060] Figure 7 is the position of the straight pipe of equal diameter and its axis in the three-dimensional space under the camera coordinate system. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. The specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0062] Example 1

[0063] like Figure 4 As shown, this embodiment exemplarily shows a straight pipe spatial posture visual tracking method based on edge contour, including the following steps:

[0064] S1. Reference Figure 1 , the two-dimensional plane target 2 is horizontally laid on the operating platform 1. The two-dimensional plane target used in this embodiment is as follows Figure 2 The two-dimensional plane is a chessboard shown in FIG. 2 , and a camera 6 is fixedly provided by a frame 5 obliquely above the two-dimensional plane target 2. The two-dimensional plane target 2 is placed in the field of view of the camera 6 and the position of the two-dimensional plane target 2 is adjusted. The camera coordinate system O-XYZ is established with the optical center O of the camera as the origin, with the X axis pointing to the right side of the camera, the Y axis pointing downward, and the Z axis pointing forward. The camera 6 is used to collect multiple two-dimensional plane target images at different positions. The camera is calibrated using the "Camera Calibration Toolbox for Matlab" toolbox developed by Jean-Yves Bouguet based on Zhang's calibration method for the multiple two-dimensional plane target images collected by the camera to obtain the camera's intrinsic parameter matrix. and distortion coefficients (k1, k2), where (f x ,f y ) represent the normalized focal length of the camera in the horizontal and vertical direction, respectively, (u0, v0) represents the principal point coordinates, and k1 and k2 represent the first and second order radial distortion coefficients, respectively.

[0065] S2, referring to Figure 3 , an equal-diameter straight pipe 4 with a radius (outer diameter of the equal-diameter straight pipe) of 8 mm and a length of 300 mm is horizontally placed above the two-dimensional plane target 2 by the support 3, and then the chessboard of the two-dimensional plane is removed, and an equal-diameter straight pipe image is collected using the camera 6; the equal-diameter straight pipe image is subjected to edge profile feature extraction using a canny edge detection algorithm (a conventional method in the field, which will not be described in detail in this embodiment), and distorted profile feature points are obtained;

[0066] S3, the distorted profile feature points are subjected to distortion correction in combination with the intrinsic matrix and distortion coefficients of the camera, and non-distorted profile feature points are obtained; the calculation formula of the distortion correction is shown in formula (1):

[0067]

[0068] wherein: and (u, v) represent the pixel coordinates of the distorted and non-distorted profile feature points, respectively; (x, y) represents the non-distorted normalized image coordinates; (k1, k2) represents the radial distortion coefficients of the camera; and (u0, v0) represents the principal point coordinates in the intrinsic matrix of the camera.

[0069] S4, the non-distorted profile feature points are subjected to straight line fitting to obtain two profile lines of the equal-diameter straight pipe;

[0070] S41, the non-distorted profile feature points are subjected to screening using the RANSAC algorithm to obtain feature point sets {(u′ g ,v′ g )|g∈1,…,G} and {(u′ m ′,v′ m ′)|m∈1,…,M} on the two profiles of the equal-diameter straight pipe, wherein (u′ g ,v′ g ) and (u′ m ′,v′ m ′) represent the pixel coordinates of the feature points on the two profiles, respectively;

[0071] S42, based on the least squares method, the feature point sets are subjected to straight line fitting to obtain two profile lines l1 and l2 of the equal-diameter straight pipe, and the equation of the fitted straight line is shown in formula (2):

[0072]

[0073] Wherein: a', b', c' and a", b", c" are the equation coefficients of two fitting straight lines respectively; the solving process of a', b', c' and a", b", c" is as follows:

[0074] The equation group respectively constituted by the homogeneous coordinates of the feature points on the two contours as the coefficient matrix is shown in formula (3):

[0075]

[0076] For formula (3), the characteristic vector corresponding to the minimum eigenvalue of the coefficient matrix is the least square solution of the equation group; or the SVD decomposition is performed on the coefficient matrix, and then the vector corresponding to the minimum singular value in the right singular matrix is taken as the least square solution;

[0077] The least square solution obtained is brought into formula (2) to obtain the two contour lines l1 and l2 of the equal-diameter straight pipeline as shown in formula (4): Figure 6

[0078] S5, combining the two contour lines of the equal-diameter straight pipeline, the radius of the equal-diameter straight pipeline and the intrinsic matrix of the camera, the spatial pose of the equal-diameter straight pipeline is solved according to the perspective projection model of the equal-diameter straight pipeline (see formula (5)); Figure 5

[0079] S51, combining the intrinsic matrix of the camera and the two contour lines l1 and l2 of the equal-diameter straight pipeline, the back projection planes corresponding to the two contour lines are respectively:

[0080]

[0081] Wherein: K represents the camera intrinsic matrix, π1 and π2 represent the back projection planes of the contour lines corresponding to l1 and l2 respectively;

[0082] S52, according to the obtained back projection planes π1 and π2, the normal vectors n1 and n2 of the back projection planes are obtained, and the normal vector of the bisector plane π m of the back projection planes π1 and π2 is:

[0083]

[0084] Wherein: n m is the bisector plane normal vector of the two back projection planes, ||n|| represents the modulus of the vector, and the included angle and intersection vector of the back projection planes π1 and π2 are respectively:

[0085]

[0086] Wherein: α is the included angle of the two back projection planes, v L ​​represents the intersection vector of the back-projection planes π1 and π2;

[0087] S53, according to the obtained intersection vector v L and the bisecting plane normal vector n m Calculate the center P of the cross section from the origin O to the support plane passing through the origin C The direction vector is:

[0088]

[0089] in: means OP C The direction vector of the straight pipe of equal diameter is combined with the radius r of the straight pipe and the angle α between the two back-projection planes to calculate the origin O and P. C The distance between points is:

[0090]

[0091] in: Indicates point O to P C The distance between the points, according to OP C Direction vector and point O and P C Distance of points Get P C The 3D coordinates of a point:

[0092]

[0093] Since the radius of the straight pipe with equal diameter is known, the spatial position of the straight pipe with equal diameter is given by P C The three-dimensional coordinates of the point and the direction vector v of the axis L OK, specifically Figure 7 shown.

[0094] Example 2

[0095] like Figure 1 、 Figure 2 、 Figure 3 As shown, this embodiment exemplarily shows a straight pipe spatial posture visual tracking device based on edge contour, including:

[0096] An operating platform 1 for laying a two-dimensional plane target 2;

[0097] The two-dimensional plane target 2 can be movably laid on the operating platform 1 to adjust multiple different positions within the camera field of view; the two-dimensional plane target used in this embodiment is as follows Figure 2 The two-dimensional plane is shown as a checkerboard;

[0098] The camera 6 is fixed above the two-dimensional plane target 2 by the frame 5, and the two-dimensional plane target 2 is always located in the camera field of view during the camera calibration;

[0099] The support 3 is arranged on the operation platform 1 at both sides of the two-dimensional plane target;

[0100] The equal-diameter straight pipe 4 is arranged on the support 3 and located in the camera field of view;

[0101] The processing system (not shown in the figure) is connected with the camera 6 and executes the above-mentioned edge contour-based straight pipe space pose visual tracking method, and the processing system is usually stored in a computer.

[0102] The device of the embodiment in combination with the edge contour-based straight pipe space pose visual tracking method of the embodiment 1 can quickly determine the pose of the equal-diameter straight pipe with a known radius and realize the pose tracking of the equal-diameter straight pipe.

[0103] The technical scheme of the present application is not limited to the above-mentioned specific embodiments, and any technical transformation according to the technical scheme of the present application falls within the protection scope of the present application.

Claims

1. A straight pipe spatial posture visual tracking method based on edge contour, characterized in that: The following steps are involved: S1. Lay a two-dimensional plane target horizontally on an operating platform, and fix a camera obliquely above the two-dimensional plane target. Place the two-dimensional plane target within the camera's field of view and adjust its position. Use the camera to capture multiple images of the two-dimensional plane target at different positions. Use Zhang's calibration method to calibrate the camera based on the multiple two-dimensional plane target images captured by the camera to obtain the camera's intrinsic parameter matrix and distortion coefficient. S2. Place a straight pipe of equal diameter with a known radius horizontally within the camera's field of view using a bracket, and use the camera to capture an image of the straight pipe of equal diameter; extract edge contour features from the image of the straight pipe of equal diameter to obtain distorted contour feature points; S3, combining the camera's intrinsic parameter matrix and distortion coefficient to perform distortion correction on the distorted contour feature points to obtain undistorted contour feature points; S4. Perform linear fitting on the undistorted contour feature points to obtain two contour lines of straight pipes with equal diameters; S5. Combining the two contour lines of the straight pipe, the radius of the straight pipe, and the intrinsic parameter matrix of the camera, the spatial pose of the straight pipe is calculated according to the perspective projection model of the straight pipe: S51. Combining the camera's intrinsic parameter matrix and the two contour lines l1 and l2 of the straight pipe with equal diameter, the back-projection planes corresponding to the two contour lines are: Where: K represents the camera intrinsic parameter matrix, π1 and π2 represent the back-projection planes of the contour lines corresponding to l1 and l2 respectively; S52, according to the obtained back-projection planes π1 and π2, obtain their normal vectors n1 and n2, and calculate the bisector π of the back-projection planes π1 and π2 m The normal vector is: Where: n m is the normal vector of the bisector of the two back-projection planes, ||·|| represents the modulus of the vector, and the angle and intersection vector of the back-projection planes π1 and π2 are calculated as follows: Where: α is the angle between the two back-projection planes, v L represents the intersection vector of the back-projection planes π1 and π2; S53, according to the obtained intersection vector v L and the bisecting plane normal vector n m Calculate the center P of the cross section from the origin O to the support plane passing through the origin C The direction vector is: in: means OP C The direction vector of the straight pipe of equal diameter is combined with the radius r of the straight pipe and the angle α between the two back-projection planes to calculate the origin O and P. C The distance between points is: in: Indicates point O to P C The distance between the points, according to OP C Direction vector and point O and P C Distance of points Get P C The 3D coordinates of a point: Since the radius of the straight pipe with equal diameter is known, the spatial position of the straight pipe with equal diameter is given by P C The three-dimensional coordinates of the point and the direction vector v of the axis L Sure.

2. The method for visually tracking the spatial position of a straight pipe based on edge contour according to claim 1 is characterized in that: In step S2, edge contour features of the image of straight pipes of equal diameters are extracted using a canny edge detection algorithm.

3. The straight pipe spatial posture visual tracking method based on edge contour according to claim 2 is characterized in that: In step S3, the calculation formula for distortion correction is shown in formula (1): in: and (u, v) represent the pixel coordinates of the contour feature points with and without distortion, respectively; (x, y) represent the normalized image coordinates without distortion; (k1, k2) represent the radial distortion coefficients of the camera; (u0, v0) represent the principal point coordinates in the intrinsic parameter matrix of the camera.

4. The method for visually tracking the spatial position of a straight pipe based on edge contour according to claim 3 is characterized in that: The step S4 comprises: S41, use the RANSAC algorithm to filter the undistorted contour feature points to obtain the feature point set {(u′ g ,v′ g )|g∈1,...,G} and {(u″ m ,v″ m )|m∈1,...,M}, where: (u′ g ,v′ g ) and (u″ m ,v″ m ) are represented as the pixel coordinates of the feature points on the two contours; S42. Based on the least squares method, the feature point set is fitted with straight lines to obtain two contour lines l1 and l2 of the straight pipe with equal diameter. The equation of the fitted line is shown in formula (2): Where: a′, b′, c′ and a″, b″, c″ are the coefficients of the equations of the two fitting lines respectively; the solution process of a′, b′, c′ and a″, b″, c″ is as follows: The equations formed by the homogeneous coordinates of the feature points on the two contours as coefficient matrices are shown in formula (3): For formula (3), the eigenvector corresponding to the minimum eigenvalue of the coefficient matrix is ​​the least squares solution of the equation system; or the coefficient matrix is ​​decomposed by SVD, and then the vector corresponding to the minimum singular value in the right singular matrix is ​​taken as the least squares solution; Substituting the obtained least squares solution into equation (2) we can obtain the two contour lines l1 and l2 of the straight pipe with equal diameter.

5. A straight pipe spatial posture visual tracking device based on edge contour, characterized in that: include: An operating platform for laying two-dimensional plane targets; A two-dimensional plane target can be moved and laid on the operating platform to adjust to multiple different positions within the camera's field of view; The camera is fixed on the upper side of the two-dimensional plane target by a frame, and the two-dimensional plane target is always within the camera's field of view during the camera calibration process; The bracket is arranged on the operating platform on both sides of the two-dimensional plane target; A straight pipe of equal diameter is mounted on a bracket and is within the camera's field of view; A processing system is connected to the camera and executes the edge contour-based straight pipe spatial posture visual tracking method as described in any one of claims 1 to 4.

6. The device for visually tracking the spatial position of a straight pipe based on edge contours according to claim 5, characterized in that: The two-dimensional plane target is a two-dimensional plane checkerboard.

Images